Client
cloud.google.com/go/bigtable v1.52.0
google.golang.org/grpc v1.83.0
Environment
Go 1.26, scratch containers on GKE
Code and Dependencies
Code is too big and ugly to list here. But it's just doing normal reads and writes to a Bigtable table
Expected behavior
Memory use doesn't keep growing
Actual behavior
Memory use does keep growing.
Screenshots
Additional context
This is very similar to github.com//issues/14582, but slower. Profiles point to similar objects being leaked. The leak is more apparent in services with low traffic - but that's perhaps because their memory usage is very small compared to the size of the leak.
Here's some profile output showing inuse_objects and inuse_space compared to a baseline profile taken about 18 hours before
Showing nodes accounting for 369747, 51.91% of 712220 total
Dropped 68 nodes (cum <= 3561)
Showing top 20 nodes out of 213
flat flat% sum% cum cum%
120152 16.87% 16.87% 120152 16.87% net.JoinHostPort (inline)
52025 7.30% 24.17% 172177 24.17% google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
32768 4.60% 28.78% 32768 4.60% google.golang.org/grpc/internal/xds/balancer/outlierdetection.newCallCounter
32768 4.60% 33.38% 32768 4.60% google.golang.org/protobuf/internal/filedesc.(*Builder).optionsUnmarshaler
21845 3.07% 36.44% -10924 1.53% google.golang.org/grpc/internal/hierarchy.SetInEndpoint
16384 2.30% 38.74% 16384 2.30% google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).switchTo
16384 2.30% 41.04% 16384 2.30% google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).populateAggregateClusterLocked
16384 2.30% 43.34% 16384 2.30% google.golang.org/grpc/internal/xds/xdsdepmgr.newClusterWatcher
10923 1.53% 44.88% 10923 1.53% fmt.Sprintf
10923 1.53% 46.41% 10923 1.53% google.golang.org/grpc/balancer/rls/internal/keys.MakeBuilderMap
6554 0.92% 47.33% 6554 0.92% google.golang.org/grpc/internal/xds/clients/xdsclient.(*AnyProto).ToAny
4681 0.66% 47.99% 4681 0.66% reflect.unsafe_New
4096 0.58% 48.56% 4096 0.58% github.com/modern-go/reflect2.newUnsafeStructField
4096 0.58% 49.14% 4096 0.58% github.com/unravelin/core/feature-extraction-service/proto.buildFlattenStruct
4096 0.58% 49.71% 4096 0.58% google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
3641 0.51% 50.23% 3641 0.51% encoding/json.(*decodeState).literalStore
3641 0.51% 50.74% 24822 3.49% encoding/json.(*decodeState).object
3344 0.47% 51.21% 3344 0.47% runtime.mallocgc
2521 0.35% 51.56% -2160 0.3% context.(*cancelCtx).propagateCancel
2521 0.35% 51.91% 2521 0.35% google.golang.org/grpc/balancer/weightedtarget/weightedaggregator.(*Aggregator).Add
Showing nodes accounting for 19.54MB, 27.57% of 70.88MB total
Dropped 35 nodes (cum <= 0.35MB)
Showing top 20 nodes out of 225
flat flat% sum% cum cum%
6.50MB 9.18% 9.18% 9MB 12.70% google.golang.org/grpc/internal/xds/xdsclient/xdsresource.parseEndpoints
2.50MB 3.53% 12.70% 2.50MB 3.53% net.JoinHostPort (inline)
1.50MB 2.12% 14.83% 1.50MB 2.12% google.golang.org/protobuf/internal/impl.consumeBytesNoZero
1MB 1.41% 16.24% 1MB 1.41% google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).switchTo
0.52MB 0.73% 16.97% 0.52MB 0.73% google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile
0.51MB 0.72% 17.69% 0.51MB 0.72% github.com/unravelin/core/lib/lang/internal/names.map.init.0
0.50MB 0.71% 18.40% 0.50MB 0.71% math/rand.newSource
0.50MB 0.71% 19.11% 0.50MB 0.71% github.com/unravelin/core/pspdomain.map.init.1
0.50MB 0.71% 19.81% 0.50MB 0.71% cloud.google.com/go/bigtable/internal/metrics.NewFactory
0.50MB 0.71% 20.52% 0.50MB 0.71% google.golang.org/grpc/balancer/weightedtarget/weightedaggregator.(*Aggregator).Add
0.50MB 0.71% 21.22% 0.50MB 0.71% google.golang.org/grpc/internal/xds/resolver.(*xdsResolver).addOrGetActiveClusterInfo
0.50MB 0.71% 21.93% 1MB 1.41% google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).applyRouteConfigUpdateLocked
0.50MB 0.71% 22.63% 1.50MB 2.12% google.golang.org/grpc/internal/xds/xdsdepmgr.(*DependencyManager).populateClusterConfigLocked
0.50MB 0.71% 23.34% 0.50MB 0.71% google.golang.org/protobuf/internal/impl.(*MessageInfo).makeFieldTypes
0.50MB 0.71% 24.05% 0.50MB 0.71% reflect.mapassign_faststr0
0.50MB 0.71% 24.75% 0.50MB 0.71% encoding/json.(*decodeState).literalStore
0.50MB 0.71% 25.46% 2.50MB 3.53% encoding/json.(*decodeState).object
0.50MB 0.71% 26.16% 0.50MB 0.71% github.com/modern-go/reflect2.newUnsafeStructField
0.50MB 0.71% 26.87% 0.50MB 0.71% github.com/unravelin/core/feature-extraction-service/proto.buildFlattenStruct
0.50MB 0.71% 27.57% 0.50MB 0.71% google.golang.org/protobuf/internal/encoding/json.(*Decoder).parseString
Client
cloud.google.com/go/bigtable v1.52.0
google.golang.org/grpc v1.83.0
Environment
Go 1.26, scratch containers on GKE
Code and Dependencies
Code is too big and ugly to list here. But it's just doing normal reads and writes to a Bigtable table
Expected behavior
Memory use doesn't keep growing
Actual behavior
Memory use does keep growing.
Screenshots
Additional context
This is very similar to github.com//issues/14582, but slower. Profiles point to similar objects being leaked. The leak is more apparent in services with low traffic - but that's perhaps because their memory usage is very small compared to the size of the leak.
Here's some profile output showing inuse_objects and inuse_space compared to a baseline profile taken about 18 hours before