fix: add pagination to deposit fetching

[Jiloc]

Description

Closes: #1148

Changes

• add pagination handling to get_deposits
• add pagination_timeout to EmilyClient - it causes the client to stop fetching deposits - but still return the ones alrealdy fetched - when the time passed > timeout.
• added pagination_timeout to the configs - currently set to 30

Testing Information

• Added integration tests to test pagination and timeout on get_deposits

Checklist:

• I have performed a self-review of my code
• My changes generate no new warnings
• New and existing unit tests pass locally with my changes

[Jiloc]

The idea of using the the BlockHeight to filter deposits already processed, may not work. The BlockHeight is not guaranteed to remain consistent in Emily (also currently they are returned from newest to oldest). In the event of a reorg, requests could be updated with a different associated block. This could result in requests being modified to have a lower height than the previous one. If we only consider requests with a BlockHeight greater than or equal to a certain value, it could lead to signers skipping requests after a reorg. A potential solution could be to add a created_at parameter to Dynamo and use that value for sorting and ignoring.

[Jiloc]

I'd propose to decide on a strategy for sorting for Emily and implement a filtering mechanism on the signers side in a separate PR

[cylewitruk]

The idea of using the the BlockHeight to filter deposits already processed, may not work. The BlockHeight is not guaranteed to remain consistent in Emily (also currently they are returned from newest to oldest). In the event of a reorg, requests could be updated with a different associated block. This could result in requests being modified to have a lower height than the previous one. If we only consider requests with a BlockHeight greater than or equal to a certain value, it could lead to signers skipping requests after a reorg. A potential solution could be to add a created_at parameter to Dynamo and use that value for sorting and ignoring.

If I got to choose, I would probably have opted for a streaming protocol backed by durable topic subscriptions (at least that's what they're called in the message broker world..). For example MQTT over Websockets (AWS version), instead of polling. That would obsolete the above challenges.

[Jiloc]

The idea of using the the BlockHeight to filter deposits already processed, may not work. The BlockHeight is not guaranteed to remain consistent in Emily (also currently they are returned from newest to oldest). In the event of a reorg, requests could be updated with a different associated block. This could result in requests being modified to have a lower height than the previous one. If we only consider requests with a BlockHeight greater than or equal to a certain value, it could lead to signers skipping requests after a reorg. A potential solution could be to add a created_at parameter to Dynamo and use that value for sorting and ignoring.

If I got to choose, I would probably have opted for a streaming protocol backed by durable topic subscriptions (at least that's what they're called in the message broker world..). For example MQTT over Websockets (AWS version), instead of polling. That would obsolete the above challenges.

I love the idea and believe this is the cleanest way to go for the signers. We'll have to design it for a future milestone though, it will require a different infra because of the usual limitations of the current setup...
We still need to support basic pagination, filtering and ordering in the REST API though. That will help in the short term with the signers and for every other user downstream (e.g. the bridge) for the long.

[djordon]

We don't need to use Emily for withdrawal requests, just for deposits. The plan was for the signers to just use the stacks node for withdrawals.

[djordon]

This one is missing the page_size config parameter. I think we should have it here, even if we have some parameters that we exclude from the docker/mainnet/sbtc-signer/signer-config.toml.in example.

[djordon]

Related to the page_size, is it supposed to specify a number of items or the number of bytes to return per page? Is there a maximum or minimum value enforced by AWS? And do we want this to be a config parameter at all?

[Jiloc]

It's the specific number of items. If the number is too big, Dynamo will always cap the query results to 1MB.

I needed to have access to it for testing purposes only, but I am unsure yet if we'll ever need to manually modify this value in the settings. I am open to remove it from the configs if you think it's better

[djordon]

Yeah, probably best to remove it from the configs since we don't plan on having folks change it.