Limit how much data we retrieve for a given CID

[bajtos]

At the moment, the SPARK node tries to retrieve all content of CID, regardless of the size. Some CIDs represent GBs of data.

IMO, this is a problem - we don't want Stations to use so much bandwidth.

It also creates a problem in spark-api, where we currently represent byte_length as a 32bit signed integer, which overflows at 2GB.

2023-09-05T15:54:19Z app[17814d5b527638] cdg [info]error: value "2753993443" is out of range for type integer
2023-09-05T15:54:19Z app[17814d5b527638] cdg [info]    at /app/node_modules/pg-pool/index.js:45:11
2023-09-05T15:54:19Z app[17814d5b527638] cdg [info]    at runMicrotasks (<anonymous>)
2023-09-05T15:54:19Z app[17814d5b527638] cdg [info]    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2023-09-05T15:54:19Z app[17814d5b527638] cdg [info]    at async setRetrievalResult (file:///app/index.js:74:5)
2023-09-05T15:54:19Z app[17814d5b527638] cdg [info]    at async handler (file:///app/index.js:12:5) 

I am proposing to introduce a new retrieval error status - content too large.

Tasks

Preview Give feedback

1. Implement the limit in SPARK module
2. Release a new version of SPARK & Station Core & Station Desktop
3. Update SPARK API to record the new measurement field flagging CARs too large
4. Update Grafana charts to handle this new flag

[juliangruber]

Spark clients should be allowed to abort retrieval if it is too large, without getting penalized. Then ideally they won't even report the result. However, Spark shouldn't have a problem with retrieval testing for large CIDs, a result is a result and is useful.

I think therefore we want the solution to be on the Station module side - it should abort the request - and for the Station module not to be penalized for not reporting in a large retrieval.