s3 worker crashing (assertion error)

[msegura]

It seems the s3 worker process might be crashing. This could be related to a metadata outage.

The stack trace is :

{"name":"S3",","error":"null == true","stack":"
AssertionError: null == true
    at process.target._send (internal/child_process.js:519:5)
    at process.<anonymous> (internal/child_process.js:485:12)
    at emitTwo (events.js:92:20)
    at process.emit (events.js:172:7)
    at internal/child_process.js:696:12
    at nextTickCallbackWith0Args (node.js:420:9)
    at process._tickCallback (node.js:349:13)
","level":"fatal","message":"caught error","hostname":"redacted","}

[ploki]

Seems related to nodejs/node#4205

[ploki]

fixed in node by this commit nodejs/node@57a51a0

this is include in node 4.5.0

[ploki]

seems to be related to #546

[farias-m]

Attached S3 log for 10.64.50.106 approx one minute prior to event at

1485099379670

Assuming that this timestamp is in milliseconds:
GMT: Sun, 22 Jan 2017 15:36:19.670 GMT
Your time zone: 1/22/2017, 7:36:19 AM GMT-8:00

Added to VPN (below)
/home/public/mc/548.minute.prior.txt

[vrancurel]

It is included in 6.0 onwards

[ploki]

@vrancurel and since v4.2.5

[LaurenSpiegel]

per discussion with @ploki there still seems to be a race condition in node around here:
https://github.com/nodejs/node/pull/4418/files#diff-f45eb699237c2e38dc9b49b588933c11R519

This should only occur for us when we are killing workers which only happens due to another problem. So this is only a secondary effect that should happen rarely.