feat: support pre caveat in store/list and upload/list
#139
Conversation
this implements support for the `prev` caveat proposed in storacha/w3up#423 I'm not sure the semantics are right yet, deploying to get a test env to play with
and use it to implement support for the `pre` caveat in `store/list` and `upload/list`
View stack outputs
|
after working with dynamo a bit, I realized that adding startCursor and endCursor to the response, combined with the new `pre` caveat gives us fully stateless bi-directional paging! I'd like to propose we deprecate `cursor` in favor of `startCursor` and `endCursor`. even if we end up moving to an implementation that doesn't support `startCursor`, `endCursor` feels a bit clearer, and we don't have as tight space constraints in these responses as we do in caveats that get included in UCANs
There was a problem hiding this comment.
Code changes look good! Thanks @travis
Can we please add some tests for this under upload-api/test/service/{store, upload}.test.js?
per the change in storacha/w3infra#139, add startCursor and endCursor and deprecate cursor
Use the new `pre` caveat and `startCursor` and `endCursor` from storacha/w3infra#139 to implement reverse paging. One unsatisfying issue with this is that paging backwards reverses the order of items in the uploads list, and that edge conditions behave fairly confusingly. Not entirely sure what to do about this yet, so pushing up for some feedback. This currently only works with a bunch of custom service config and `file:/` dependencies that I am not pushing up for now.
1) cleaner endCursor handling 2) add deprecation comments
Ah yes - will jump on this tomorrow most likely! |
|
ok @vasco-santos got some tests in, please take a look! two questions came out of this: |
|
We synced offline in a, should be good to go with Docker installed locally. But maybe there is a requirement so have some environment variables AWS specific setup (per CI file). b) we can do in follow up per discussion in sync later |
per the change in storacha/w3infra#139, add startCursor and endCursor and deprecate cursor
Use the new `pre` caveat and `before` and `after` cursors from storacha/w3infra#139 to implement reverse paging in the uploads list headless components and w3console https://user-images.githubusercontent.com/1113/218706799-a0cac0cd-7da2-47d5-b19e-a2bc41ceda2f.mov
per the change in storacha/w3infra#139, add startCursor and endCursor and deprecate cursor
Use the new `pre` caveat and `before` and `after` cursors from storacha/w3infra#139 to implement reverse paging in the uploads list headless components and w3console https://user-images.githubusercontent.com/1113/218706799-a0cac0cd-7da2-47d5-b19e-a2bc41ceda2f.mov
Implement support for the
precaveat introduced in storacha/w3up#423Also introduce
startCursorandendCursorwith the intention to deprecatecursor. By providing cursors for the beginning and end of the range, we enable clients to page forward or backward without saving state anywhere.Example usage:
Given uploads like:
You can now page forward like:
and page backward like:
Note that the cursor values here happen to match the CID strings - this is an implementation detail and should not be relied upon. Instead, cursor values should be taken from
startCursorandendCursorin the list response:$ w3 can upload ls --size 4 --cursor bafybeidibxjgswcu2q4gx7j5mzq26qtcczt3nczg6k4ybqct2rm567kpoy --raw {"size":4,"cursor":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq","results":[{"root":{"/":"bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji"},"shards":[{"/":"bagbaieracmkgwrw6rowsk5jse5eihyhszyrq5w23aqosajyckn2tfbotdcqq"}],"updatedAt":"2023-02-13T16:30:04.172Z","insertedAt":"2023-02-13T16:30:04.172Z"},{"root":{"/":"bafybeiefiddap3p3z3p3obatviipgda2us3umt4uxeh2trrni5pt6bfc6e"},"shards":[{"/":"bagbaieraz3iyicghnyzpjld7yxf7rmntf5cdbewo4pmuck6kkkc6f7z3sznq"}],"updatedAt":"2023-02-13T16:29:51.864Z","insertedAt":"2023-02-13T10:57:39.017Z"},{"root":{"/":"bafybeifohbynkellbberk3hi447dntungwgicmsvxuwdttk5k5k535mw2a"},"shards":[{"/":"bagbaierasu5cufqagk34dkmhszbwuhiydjjjp3pl4bacyvskljqtkcxhgbta"}],"updatedAt":"2023-02-13T16:30:11.648Z","insertedAt":"2023-02-13T10:57:16.264Z"},{"root":{"/":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq"},"shards":[{"/":"bagbaieralqhbrq2mqupqijkoylagbams263jwq6helsl6kbdpivupjm7egxq"}],"updatedAt":"2023-02-13T16:30:18.403Z","insertedAt":"2023-02-13T10:57:49.695Z"}], "endCursor":"bafybeigkeedelu7fkqgeafzh7zrtwmcn747clncxuufl2dzu46mww735qq","startCursor":"bafybeie7vavzb2ojufppuaxer3kfu4yxi36z7ozdz7wnhtylnd7rl5p6ji"}nb that these w3 commands require the changes I implemented in storacha/w3cli#45