[Merged by Bors] - v2alpha1 atxs stream #5566
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # api/grpcserver/config.go # go.mod # go.sum # node/node.go # sql/atxs/atxs.go
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## develop #5566 +/- ##
=========================================
- Coverage 79.7% 79.7% -0.1%
=========================================
Files 267 270 +3
Lines 26857 27186 +329
=========================================
+ Hits 21416 21669 +253
- Misses 3920 3970 +50
- Partials 1521 1547 +26 ☔ View full report in Codecov by Sentry. |
kacpersaw
requested review from
dshulyak,
fasmat,
poszu and
ivan4th
as code owners
poszu
reviewed
Feb 16, 2024
Co-authored-by: Bartosz Różański <[email protected]>
poszu
approved these changes
Feb 16, 2024
|
bors merge |
spacemesh-bors bot
pushed a commit
that referenced
this pull request
Feb 16, 2024
This PR is based on #5233 api: spacemeshos/api#305 - [x] watch support. subscribe to atxs emitted after validating and saving database - [x] encode the rest of the fields - [x] add support for limit and offset and make use of it in List api. public services should not expose Stream, as it is much harder to track resources for streams - [x] extract into separate service that has access only for to db pointer, in order to enable readonly replicas - [x] always order by epoch asc, id - [x] matcher for watcher based on request parameters - [x] extract query builder into separate module - [x] test coverage - [x] activations count --- ### streaming full atxs stream all atxs before and including publish epoch 2: > grpcurl -plaintext -d '{"end_epoch": 2}' 127.0.0.1:9093 spacemesh.v2.ActivationStreamService.Stream stream all atxs that belong to the specific smesher > grpcurl -plaintext -d '{"node_id": "MaAcOOLRj0pffyWO8gfuuWiBszQG2ueq/SNbbMnHHoU="}' 127.0.0.1:9093 spacemesh.v2.ActivationStreamService the same as above, but also continue waiting for atxs from same smesher > grpcurl -plaintext -d '{"node_id": "MaAcOOLRj0pffyWO8gfuuWiBszQG2ueq/SNbbMnHHoU=", "watch": true}' 127.0.0.1:9093 spacemesh.v2.ActivationStreamService.Stream ### listing atxs streaming if more convenient to use (less memory footprint, can read whole collection from same stream), however it is harder to enforce fair distribution of resources for multiple concurrent users. for public api it is better to use req/resp pattern with strict limits and no ability to continuously watch collections. list full atxs > grpcurl -plaintext -d '{"start_epoch": 2, "limit": 100, "offset": 100}' 127.0.0.1:9092 spacemesh.v2.ActivationService.List note that limit and offset are effective only for specific query, so that if you started iteration with start_epoch: 2, you can't just bump start_epoch to 3 and continue using same offset and limit if you care about getting correct results. ### activations count per epoch > grpcurl -plaintext -d '{"epoch": 2}' 127.0.0.1:9092 spacemesh.v2alpha1.ActivationService.ActivationsCount ### versioning data in spacemesh is mostly static and not expected to change constantly. for the case when it does change, object can be bumped to the next version. all responses return protobuf oneof version type. Co-authored-by: Dmitry <[email protected]>
|
Pull request successfully merged into develop. Build succeeded:
|
spacemesh-bors
bot
changed the title
fasmat
reviewed
Feb 20, 2024
| @@ -36,7 +36,7 @@ require ( | |||
| github.com/quic-go/quic-go v0.41.0 | |||
| github.com/santhosh-tekuri/jsonschema/v5 v5.3.1 | |||
| github.com/seehuhn/mt19937 v1.0.0 | |||
| github.com/spacemeshos/api/release/go v1.28.0 | |||
| github.com/spacemeshos/api/release/go v1.28.1-0.20240215101325-80fe2752264b | |||
There was a problem hiding this comment.
@kacpersaw could you update this to the released version (v1.29.0)?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is based on #5233
api: spacemeshos/api#305
streaming full atxs
stream all atxs before and including publish epoch 2:
stream all atxs that belong to the specific smesher
the same as above, but also continue waiting for atxs from same smesher
listing atxs
streaming if more convenient to use (less memory footprint, can read whole collection from same stream), however
it is harder to enforce fair distribution of resources for multiple concurrent users. for public api it is better to use req/resp pattern
with strict limits and no ability to continuously watch collections.
list full atxs
note that limit and offset are effective only for specific query, so that if you started iteration with start_epoch: 2, you can't just bump start_epoch to 3 and continue using same offset and limit if you care about getting correct results.
activations count per epoch
versioning
data in spacemesh is mostly static and not expected to change constantly. for the case when it does change, object can be bumped to the next version. all responses return protobuf oneof version type.