Accounts for Signer endpoint in Horizon using new ingestion system #1472
Comments
Good question. I actually forgot to mention that this new feature should by behind a feature flag (default=off). So in the worst case we can just include a note that new feature requires ~4GB of RAM.
OK, will add a new point. |
This was referenced Jul 12, 2019
bartekn
added a commit
that referenced
this issue
Jul 18, 2019
The goal of this commit is to add a new endpoint that presents "Accounts For Signer" data (close #1486) and integrate all features listed in #1472 to actually process the data and expose it. This required us to add a new `expingest` package to Horizon that is completely independent of the old `ingest` package and is using the new `exp/ingest` system. The pipeline currently saves signers data to the DB but can be easily extended with more features in the future. The new endpoint (`/accounts`) is future-proof. It currently works only with `signer` GET parameter set but when more processors are added to the ingestion pipeline we can add new params that will allow filtering accounts by assets, by home domain, by inflation destination etc. When the rest of Horizon is migrated to the new system, `/accounts` will be a collection of Account Details resources (like in `/accounts/{id}`). Known limitations & issues: - Enabling this feature requires around 4GB of RAM for processing initial state and a space for around 3M rows in `accounts_signers` table for pubnet. - The endpoint will not be available until the state from history archives is not fully parsed. This usually takes a couple minutes. Before then we display `StillIngesting` problem in `/accounts`. - New ingestion system is independent of the old one so new endpoint data may not be up to date with the rest of the data in Horizon.
|
It's live on horizon-testnet: Please check it out . |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We're going to test new
exp/ingestpackage by creating a simple "Accounts for Signer" endpoint in Horizon (#432). The new ingestion system will be completely separated from an existing ingestion of Horizon. There will be no synchronization so it's possible that "Accounts for Signer" data will not be presenting data at the same ledger as other Horizon endpoints.Here's a proposed list of steps to achieve this is below. Each step will be moved to another issue once the plan is approved.
LiveSessionshould update cursor value in HorizonLiveSession(in exp/ingest: Ingest Session #1456) and create pipelines that pull data from history archives and ledger backend and put them in a database. This is mostly done inhorizon-demoin exp/ingest: Ingest Session #1456.@ire-and-curses @tamirms would you like to add/modify something?
The text was updated successfully, but these errors were encountered: