Error: socket hang up and Error: read ECONNRESET

[PatrickJS]

The error seems to be with the new-relic's use of continuation-local-storage

here is the sentry report for Error: socket hang up

here is the sentry report for Error: read ECONNRESET

new-relic version 1.3.2

[othiym23]

So you're right that continuation-local-storage is involved (most likely), but since CLS and async-listener are involved in every asynchronous transaction, there's more to it than that. Are you using outbound client requests in your application? How are you handling errors? At first glance, this appears to be a duplicate of #120, and the same questions need to be answered. We have a lot of tests around New Relic for Node and CLS, and haven't yet been able to track down a reproducible case that exhibits this bug. We will be forever grateful to whomever helps us nail this one down!

[bausmeier]

I am also getting a socket hang up error after my app has been running for a few minutes:

New Relic for Node.js was unable to start due to an error:
Error: socket hang up
    at createHangUpError (http.js:1472:15)
    at CleartextStream.socketCloseListener (http.js:1522:23)
    at CleartextStream.EventEmitter.emit (events.js:117:20)
    at tls.js:696:10
    at /home/brett/nr-test/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:177:31
    at process._tickDomainCallback (node.js:459:13)
    at process.<anonymous> (/home/brett/nr-test/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/index.js:18:15)

I was trying to add New Relic to an existing app, but get the same thing when using the getting started example from the Hapi repo:

require('newrelic');
var Hapi = require('hapi');

// Create a server with a host and port
var server = Hapi.createServer('localhost', 8000);

// Add the route
server.route({
    method: 'GET',
    path: '/hello',
    handler: function (request, reply) {
        reply('hello world');
    }
});

// Start the server
server.start();

Am I missing something?

[othiym23]

@bausmeier I'm pretty sure you're seeing a different issue. Is your app crashing in addition to New Relic for Node not being able to start up? If not, this is something getting in the way of your app being able to establish an HTTPS connection to New Relic's servers. Does this happen every time?

If you can't troubleshoot what's up given that clue, set the log level of the app to trace, gzip the results, and send them to [email protected] and we'll see what we can figure out from there.

[bausmeier]

It's not crashing the app, but it does happen every time. I thought it might be something getting in the way of HTTPS, just can't figure out what. I will dig some more and send a mail if I don't get it sorted. Thanks.

[nikilster]

I think I'm getting this also

[othiym23]

Just to update everyone, we've been looking into this issue and haven't been able to reproduce it reliably in our own tests. We haven't even figured out if the issue is with our code or with something in New Relic's infrastructure yet. ;) As soon as we figure out what's going on with this, we'll be able to move forward, so if you've come up with any additional details you think would help, please let us know!

@bausmeier @nikilster @gdi2290 What version of Node are all of you running?

[nikilster]

I'm running the Heroku new relic addon. Not sure exactly what node version it is (do you know how to check?) - my package.json specifies

"engines": {
    "node": "0.10.x"
  }

so I'm assuming I'm running the latest.

[othiym23]

@bausmeier @gdi2290 It seems that @niklister is seeing this problem while running on Heroku -- where are the two of you deploying your apps?

[PatrickJS]

@othiym23 yup it's on heroku

[bausmeier]

@othiym23 Mine was just an HTTPS issue like you said. It's working fine now running on Digital Ocean.

[mintrigue]

I'm seeing it on heroku

[rarkins]

Seen on AWS Elastic Beanstalk/node.js just now:

New Relic for Node.js was unable to start due to an error:
Error: socket hang up
    at createHangUpError (http.js:1472:15)
    at CleartextStream.socketCloseListener (http.js:1522:23)
    at CleartextStream.EventEmitter.emit (events.js:117:20)
    at tls.js:696:10
    at /var/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:177:31
    at process._tickDomainCallback [as _tickCallback] (node.js:459:13)

[tyscorp]

Started getting this issue today on Digital Ocean after upgrading to 1.4.0.

[cgould]

I'm also seeing this on heroku. It is intermittent and has happened about about 20 times over the last 3 days. Here's the full error from the heroku log:

Apr 10 06:45:36 android-opentable app/web.1: {"name":"newrelic","hostname":"a8551c09-08c6-4b12-bc80-53aebae54aeb","pid":2,"component":"collector_api","level":50,"err":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at createHangUpError (http.js:1472:15)\n at CleartextStream.socketCloseListener (http.js:1522:23)\n at CleartextStream.EventEmitter.emit (events.js:117:20)\n at tls.js:696:10\n at /app/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:177:31\n at process._tickCallback (node.js:415:13)","code":"ECONNRESET"},"msg":"Calling metric_data on New Relic failed unexpectedly. Data will be held until it can be submitted:","time":"2014-04-10T13:45:36.183Z","v":0}

[groundwater]

We have released version 1.5.0 of the newrelic module, which should address this issue.

We created a new keep-alive agent called yakaa which would re-use a single SSL connection to the New Relic servers when sending metric data. Every harvest cycle, we do a few https connection to report on transaction traces, errors, and metric data. Re-using the connection keeps overhead down on both the server and client.

It looks like yakaa had a timeout bug, where it would prematurely close the connection if it was taking too long. This has been fixed in yakaa 1.0.0.

I hope you can give version 1.5.0 and let us know if your socket hangup woes have been fixed. Sorry we caused your app problems, but thank you for working with us to narrow down and reproduce this error.

I will close this issue in a few days if all seems well.

[morwalz]

i still see this error even if we are using latest npm version.
13:06:07.976 2014-05-21 07:36:07.671884+00:00 app web.3 - - {"name":"newrelic","hostname":"93451f12-6534-4344-b68d-b98319f21e28","pid":2,"component":"collector_api","level":10,"err":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at createHangUpError (http.js:1412:15)\n at CleartextStream.socketCloseListener (http.js:1462:23)\n at CleartextStream.EventEmitter.emit (events.js:117:20)\n at tls.js:616:10\n at /app/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:177:31\n at process._tickDomainCallback as _tickCallback","code":"ECONNRESET"},"msg":"Error 1 during metric_data:","time":"2014-05-21T07:36:07.671Z","v":0} {"name":"newrelic","hostname":"93451f12-6534-4344-b68d-b98319f21e28","pid":2,"component":"collector_api","level":50,"err":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at createHangUpError (http.js:1412:15)\n at CleartextStream.socketCloseListener (http.js:1462:23)\n at CleartextStream.EventEmitter.emit (events.js:117:20)\n at tls.js:616:10\n at /app/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:177:31\n at process._tickDomainCallback as _tickCallback","code":"ECONNRESET"},"msg":"Calling metric_data on New Relic failed unexpectedly. Data will be held until it can be submitted:","time":"2014-05-21T07:36:07.671Z","v":0} {"name":"newrelic","hostname":"93451f12-6534-4344-b68d-b98319f21e28","pid":2,"level":30,"err":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at createHangUpError (http.js:1412:15)\n at CleartextStream.socketCloseListener (http.js:1462:23)\n at CleartextStream.EventEmitter.emit (events.js:117:20)\n at tls.js:616:10\n at /app/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:177:31\n at process._tickDomainCallback as _tickCallback","code":"ECONNRESET"},"msg":"Error on submission to New Relic (data held for redelivery):","time":"2014-05-21T07:36:07.672Z","v":0}

[txase]

@morwalz Thanks for letting us know that things are not working for you. Since the original issue has been resolved, please open a new issue so we can track it separately.

Thanks!