[Node.js] Excessive memory consumption when large data is stored on IncomingRequest object

[lforst]

Description

A user was reporting 2x-3x memory consumption when using the Node.js SDK. I had a chat with them and will summarize the setup and findings here.

• SDK setup is a very ordinary express setup with mostly SDK defaults (traces sample rate: 0.5, 0.1 - doesn't matter, leaks more or less equally)
• Requests to the service are transient
• No long-running requests or anything
• Large objects are stored on the res object (in particular large array of strings containing identifiers)
• The memory snapshot shows that NonRecordingSpans and Spans are holding references to the res object via the _isolationScope field.
• The majority of these references seems to come from NonRecordingSpans.
• Interestingly, and I don't know if this is somehow relevant, the NonRecordingSpans are held by very deep chains of Timeout objects. The user does not define any timeouts manually.

My hypothesis: We are not cleaning up NonRecordingSpans "naturally" and keeping them in the exporter or somewhere and they end up being evicted by a timeout policy. This may be in our code, or this may be in OTELs code.

[mydea]

My best guess (slash hope) is, that this is fixed by #14806, by stopping to put request on the SDK processing metadata.

we link the scope to a span, so while that is around the request will also stay around. Not 100% sure if/where/why the spans themselves keep around, but hopefully that PR will make these things go away generally by not keeping a reference to the request around.