Repository navigation
Accumulating io.grpc.netty....MpscArrayQueue objects using google-cloud-monitoring 0.40/0.41 #3167
Description
I'm not sure if this is a bug or not, but the behavior is strange and I'm not sure if it's me misusing a library but I have a Spring scheduled task that runs periodically on a GCE instance.
When google-cloud-monitoring was added (version 0.40.0-beta initially but later 0.41.0-beta) we began making calls to write timeseries data.
This results in a large pile up over time of io.grpc.netty.shaded.io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueue objects and then eventually to an out of memory exception.
These objects also increase the presence/footprint of generic java.lang.Object instances which point to these queue objects.
We also use Datastore in this project, but these objects do not show up if I remove the monitoring piece (even though I thought Datastore also used netty/grpc for communication.)
The relevant pom.xml is:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.thrift</groupId>
<artifactId>libthrift</artifactId>
<version>0.11.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>23.6-jre</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler</artifactId>
<version>4.1.17.Final</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>2.0.7.Final</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.5</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>com.akamai</groupId>
<artifactId>NetStorageKit</artifactId>
<version>3.6.6</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.11.2</version>
</dependency>
<!-- google cloud -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>23.6-jre</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-datastore</artifactId>
<version>1.16.0</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-monitoring</artifactId>
<version>0.41.0-beta</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-logging</artifactId>
<version>1.17.0</version>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-cloudkms</artifactId>
<version>v1-rev26-1.23.0</version>
<exclusions>
<exclusion>
<artifactId>guava-jdk5</artifactId>
<groupId>com.google.guava</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>1.14.0</version>
</dependency>
<!-- end: google cloud -->
<!-- test dependencies -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.15.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.truth</groupId>
<artifactId>truth</artifactId>
<version>0.39</version>
<scope>test</scope>
</dependency>
<!-- end test dependencies -->
</dependencies>
We are using java-1.8.0-openjdk.x86_64 on a Centos 7 build, deployed in GCE as a Docker container. (With yum-plugin-ovl installed to get around another Google monitoring issue.)
The code to write the timeseries data is:
/**
* Method to write a IMetricPoint to Stackdriver, does most of the driving for the class.
*
* @param metricPoint a metric type to write, defined as a constant on this class.
* @param labels a string map of metric labels
*/
public void writePoint(IMetricPoint metricPoint, Map<String, String> labels) {
Metric metric = Metric.newBuilder()
.setType(String.format("custom.googleapis.com/%s", metricPoint.getKey()))
.putAllLabels(labels)
.putLabels(ENVIRONMENT_LABEL, config.getEnvironment())
.build();
// Google recommends only one point per timeseries, or limiting writing more
// than one to only once a minute.
// https://cloud.google.com/monitoring/custom-metrics/creating-metrics#writing-ts
ArrayList<Point> pts = new ArrayList<>();
pts.add(metricPoint.getPoint());
MonitoredResource resource = getMonitoredResource();
// by default, a GAUGE metric.
TimeSeries timeSeries = TimeSeries.newBuilder()
.setMetric(metric)
.addAllPoints(pts)
.setResource(resource)
.build();
// We could accumulate these and just have a batch job send them, eventually.
ArrayList<TimeSeries> tsList = new ArrayList<>();
tsList.add(timeSeries);
CreateTimeSeriesRequest request = CreateTimeSeriesRequest.newBuilder()
.setName(String.format("projects/%s", config.getProjectId()))
.addAllTimeSeries(tsList)
.build();
try {
MetricServiceClient client = MetricServiceClient.create();
client.createTimeSeries(request);
} catch (IOException | ApiException e) {
logError("Unable to write Stackdriver metric "
+ metricPoint.getKey()
+ e.getMessage());
}
}
/**
* Creates a resource type indicating where this metric was recorded.
*
* When deployed, a start up script should pull the GCE metadata and provide
* the zone and instance_id as environmental variables. Otherwise, we just put
* the metrics in the global zone (e.g. when running from a development box.)
*
* We cannot say our resource is a "laptop" but note that when we write a point out,
* we add the label "environment" in `writePoint` which is gotten from config and
* will read "dev" so we won't just have global metrics floating about we can't find
* the origin for.
*
* @return a resource
*/
private MonitoredResource getMonitoredResource() {
String instanceId = this.systemConfig.getenv(SystemConfig.ENV_GCE_INSTANCE_ID).orElse("");
String zone = this.systemConfig.getenv(SystemConfig.ENV_GCE_ZONE).orElse("");
if (!instanceId.isEmpty() && !zone.isEmpty()) {
return MonitoredResource.newBuilder()
.setType(GCE_RESOURCE_TYPE)
.putLabels(INSTANCE_ID_LABEL, instanceId)
.putLabels(ZONE_LABEL, zone)
.build();
} else {
return MonitoredResource.newBuilder()
.setType(GLOBAL_RESOURCE_TYPE)
.build();
}
}In this case config is simply a Spring component that can be auto wired in with some environmental config data and systemConfig is just a functional wrapper around System.getenv.
I have also called this code without the auto-wiring and I still see these objects created.
Any pointers would be great, hoping this is just misuse of the library somehow.
Activity
pongad commentedon Apr 18, 2018
Hi @Dan-Ostrowski-CK . I think there are a couple of things you can change.
client.close()so not calling it is probably the cause of the memory consumption. I recommendclientis safe for concurrent use, so maybe a more effective way is to create the client only once and use it across many calls towritePoint. This will probably also make your client faster, since setting up network connections and auth can be amortized over many calls.I'm pretty sure that doing the above should fix the problem, so I'll close the issue. If it doesn't, please feel free to reopen.
docs: Add samples for using timestamps with BQStorage Read and Write …
chore(main): release 2.38.1 (#3167)
docs: Add samples for using timestamps with BQStorage Read and Write …
chore: update googleapis commit at Thu Sep 5 02:21:55 UTC 2024 (#3167)
fix: do not end transaction span when rolling back to savepoint (#3167)
chore: merge train for e2e-crc32c-media related upload PRs (#3197)