perf(dapi)!: replace getMnListDiff with a streaming endpoint
#1859
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shumkov
changed the title
shumkov
changed the title
# Conflicts: # packages/dapi/lib/rpcServer/server.js
shumkov
changed the title
getMnListDiff with a streaming endpoint
pshenmic
previously approved these changes
May 30, 2024
# Conflicts: # packages/dapi-grpc/clients/core/v0/java/org/dash/platform/dapi/v0/CoreGrpc.java # packages/dapi-grpc/clients/core/v0/python/core_pb2.py
…at/dapi/mnlist-stream
shumkov
changed the title
getMnListDiff with a streaming endpointgetMnListDiff with a streaming endpoint
markin-io
approved these changes
Jun 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
Currently, the
getMnListDiffendpoint allows to fetch masternode diff (dip-4) withing range of blocks. It uses theprotx diffCore RPC which is very slow. DAPI client requests the initial list from genesis block to tip and then request the difference with new tip every several seconds.To reduce load on the Core and network, we need to implement a streaming endpont that will keep masternode list and updates globaly for all client requests in DAPI. It will return full mn list from genesis to tip as a first message and update as sepatate messages every new block.
What was done?
getMnListDifffrom DAPI and DAPI ClientsubscribeToMasternodeListendpoint to DAPI with global state to prevent extra calls to Core RPCHow Has This Been Tested?
During the testing, we noticed that on CI (probably the specific browser version for linux is affected) one of the streams stopped receiving messages. It's quite an old Brave browser from 2022. We weren't able to reproduce it locally or in other environments. A theory was that it was because of the number of open streams but it wasn't the case. At least not directly. To solve this problem I would recommend upgrading gRPC stack which is super outdated. gRPC Web library was updated in the 2021 year. The goal of this PR is not to fix the existing gRPC Web problem but to protect the DAPI endpoint. We skipped a couple of tests to be run in a browser and it helped for some reason.
Breaking Changes
getMnListDiffendoint is removedChecklist:
For repository code-owners and collaborators only