Agent version 8.16.0 has connection issues to NR

[pvarga88]

Description

We were using Java agent version 8.15.0 and had no issues. With the only change being an upgrade to 8.16.0, our logs are now full of these messages:

com.newrelic WARN: IOException (message: Connection reset, cause: null) while trying to send data to New Relic. EventBatch retry recommended
com.newrelic INFO: [EventBatch] - Batch sending failed. Backing off 0 MILLISECONDS
com.newrelic WARN: IOException (message: Connection reset, cause: null) while trying to send data to New Relic. EventBatch retry recommended
com.newrelic INFO: [EventBatch] - Batch sending failed. Backing off 0 MILLISECONDS

Also, presumably due to buffering/queueing of metrics, the heap usage gradually goes up to 500 MB.
This is accompanied by messages like so

com.newrelic WARN: Refusing to schedule batch of size 463 (would put us over max size 1000000, available = 410)

Your Environment

Java OpenJDK 17.0.13+11
Using G1 GC

Additional context

Just skimming through the release notes, I would suspect this change is the culprit:
#2082

[workato-integration]

https://new-relic.atlassian.net/browse/NR-350762

[meiao]

The change you mention is unlikely to have affected the agent.
It is more likely that the update of JFR caused this.
#2129

Well, I assume that JFR is on.
What are the settings you are using? Do you have a proxy? Does it use HTTPS?

[pvarga88]

We specify a proxy via NEW_RELIC_PROXY_HOST/NEW_RELIC_PROXY_PORT.
We are not passing -XX:+FlightRecorder to the JVM, if that's what you mean by JFR?

[meiao]

No, I mean the JFR/Real-time profiling functionality in the Java agent.
https://docs.newrelic.com/docs/apm/agents/java-agent/configuration/java-agent-configuration-config-file/#jfr-real-time-profiling

Does your proxy require username/password?
Do you specify the schema (HTTP/HTTPS)?

I am also assuming that most of the agent works. That is, when using New Relic's UI, you can see everything, except for Real-time profiling.

[pvarga88]

Ah, yes, real-time profiling is enabled, and you're correct that's it's broken.

We aren't specifying a proxy_scheme but it seems to work anyway. I don't know what it defaults to. There's no proxy authentication.