Unhandled ECONNRESET crashes app

[jessefulton]

I am consistently getting the following error which crashes my whole app:

events.js:72 
    throw er; // Unhandled 'error' event 
                ^ 
Error: read ECONNRESET 
    at errnoException (net.js:904:11) 
    at TCP.onread (net.js:558:19) 
    at TCP.<anonymous> (/app/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:177:31) 

It appears that the source of this call can be from any number of external calls I'm making, but the log always shows New Relic as the culprit. As soon as I disable the New Relic module, things run smoothly. I've looked at #75, but it doesn't seem to explain much. Is there something specific I should be looking out for? I'm using restify to make the majority of my external API calls (both over http and https)...

I'm on node v0.10.26 and using newrelic v1.3.1.

Below is a list of my dependencies - would any of these conflict with new relic?

  "dependencies": {
    "coffee-script": "~1.6.3",
    "async": "~0.2.10",
    "bluebird": "~1.0.7",
    "winston": "~0.7.2",
    "natural": "~0.1.24",
    "node-restify-swagger": "https://github.com/jessefulton/node-restify-swagger/tarball/fix-response-messages",
    "node-restify-validation": "~0.0.5",
    "restify": "2.6.1",
    "node-uuid": "~1.4.1",
    "twit": "~1.1.11",
    "knox": "~0.8.9",
    "zencoder": "~0.4.4",
    "ngrok": "https://github.com/bubenshchykov/ngrok/tarball/master",
    "oauth": "~0.9.10",
    "newrelic": "~1.3.1",
    "amqp-rpc": "https://github.com/demchenkoe/node-amqp-rpc/tarball/master",
    "memjs": "~0.8.3",
    "redis": "~0.10.0",
    "hiredis": "~0.1.15",
    "eventemitter2": "~0.4.13",
    "amqp": "~0.1.8",
    "optimist": "~0.6.0",
    "mongoose": "~3.8.1",
    "mongoose-stamp": "0.0.2",
    "mongoose-findorcreate": "~0.1.2",
    "heroku-client": "~1.0.0",
    "analytics-node": "~0.5.0",
    "decay": "~1.0.5",
    "bitly": "https://github.com/jessefulton/node-bitly/tarball/feat/custom-domains",
    "mongoose-shortid": "~0.4.1",
    "moment": "~2.5.1",
    "elasticsearchclient": "~0.5.3",
    "memwatch": "~0.2.2",
    "busboy": "~0.2.1",
    "knox-mpu": "~0.1.6",
    "walk": "^2.3.1",
    "semver": "^2.2.1"
  },

[othiym23]

Thanks for the detailed report!

Regardless of whether or not New Relic is ultimately at fault, we're going to show up in pretty much all your asynchronous stacktraces, due to how New Relic's transaction tracer works. The fault here is in how those errors are causing your process to crash. It probably is a bad interaction between something in your app and one of New Relic's dependencies, but there's nothing that jumps out at me in your dependencies list as being a problem.

I will say that you're one version behind, and 1.3.2 includes a substantial rewrite of the connection layer between newrelic and New Relic's servers. That said, it's pretty unlikely this would make any difference to outbound requests from your application.

You might try boosting the logging label to trace and seeing if anything interesting shows up in the logs. Beyond that, I would really like to fix this problem, but right now I don't really have any good ideas as to what the cause could be. That's not exactly the most informative stacktrace!

[jessefulton]

What exactly should I be looking for in the traces?

Here are the logs around one of the crashes: looks like a mongodb disconnect? Although I haven't seen that before...
http://pastebin.com/Fb293f1J

Here is another one after I tried to restart...
http://pastebin.com/N6LQZVbx

[othiym23]

I think the first thing to do is to determine if there's a common origin for these errors so we can start trying to put together some test cases to try to reproduce this. How are you handling errors on client requests right now?

[txase]

Hi @jessefulton,

We've done a lot to improve how the agent connects to New Relic. We believe your issue may be resolved at this point. Please upgrade to the latest version of the agent to see if it works better for you.

I am going to close this issue now, but please feel free to reopen it if you continue to see issues. You may notice temporary connection issues from time to time due to network congestion, but the agent saves data and reports it the next time it is able to connect. If you experience continued connection failures in a short period of time, or anything that harms the execution of your app, please let us know.

Thanks!