mc-watcher keeps log of attestation verification reports #694
Conversation
| .expect("Could not sync signatures"); | ||
| let watcher = Watcher::new(watcher_db.clone(), transactions_fetcher, logger.clone()); | ||
|
|
||
| let _verification_reports_collector = VerificationReportsCollector::new( |
There was a problem hiding this comment.
For now we only collect verification reports inside the mc-watcher binary. mc-mobilecoind remains unchanged (it uses the WatcherDB and WatcherSyncThread to sync timestamps and block signatures but has no need for the verification reports).
|
When testing locally I used the following and the following command: |
| pub fn sync_signatures( | ||
| &self, | ||
| start: u64, | ||
| max_block_height: Option<u64>, | ||
| ) -> Result<(), WatcherError> { | ||
| ) -> Result<bool, WatcherError> { |
There was a problem hiding this comment.
This is part of a bugfix in main.rs that caused the loop to run indefinitely even when a max block was specified.
watcher/src/bin/main.rs
Outdated
| use structopt::StructOpt; | ||
|
|
||
| const SYNC_RETRY_INTERVAL: Duration = Duration::from_secs(1); |
There was a problem hiding this comment.
Why is this set here as opposed to either from config, or via using something like the retry crate?
There was a problem hiding this comment.
Its there because realistically no-one will ever choose a different value. retry is inappropriate since this is actually a polling interval so the name is misleading. I'll add it to the command line config.
watcher/src/config.rs
Outdated
| struct SourceConfig { | ||
| /// URL to use for pulling blocks. | ||
| /// | ||
| /// For example: https://s3-us-west-1.amazonaws.com/mobilecoin.chain/node1.master.mobilecoin.com/ |
There was a problem hiding this comment.
Would maybe put testnet URL as example
watcher/src/watcher_db.rs
Outdated
| @@ -40,6 +52,17 @@ impl MetadataStoreSettings for WatcherDbMetadataStoreSettings { | |||
| /// Block signatures database name. | |||
| pub const BLOCK_SIGNATURES_DB_NAME: &str = "watcher_db:block_signatures"; | |||
|
|
|||
| /// VerificationReports database name. | |||
| pub const VERIFICATION_REPORTS_BY_BLOCK_SIGNER_DB_NANE: &str = | |||
There was a problem hiding this comment.
| pub const VERIFICATION_REPORTS_BY_BLOCK_SIGNER_DB_NANE: &str = | |
| pub const VERIFICATION_REPORTS_BY_BLOCK_SIGNER_DB_NAME: &str = |
| /// This is needed because LMDB limits the value size in DUP_SORT databases to 511 bytes, not | ||
| /// enough to fit the report. This database needs to be DUP_SORT since we want to support the | ||
| /// odd case of different reports showing up for the same signer/url pair. It shouldn't happen, | ||
| /// but we sure don't want to miss it if it does. |
There was a problem hiding this comment.
Thank you for documenting this
watcher/src/watcher_db.rs
Outdated
| } | ||
|
|
||
| /// Get verification reports seen for a specific block signer/URL pair. | ||
| /// In theory there should ever be a single report (or none) for a given block_signer+src_url |
There was a problem hiding this comment.
| /// In theory there should ever be a single report (or none) for a given block_signer+src_url | |
| /// In theory there should only ever be a single report (or none) for a given block_signer+src_url |
Motivation
We'd like
mc-watcherto support more thorough watching of the network. The first step in a series of upcoming improvements is to have it fetch theVerificationReporteach time a new block signer is encountered, and store that in its database.In this PR
mc-watchersources are now configured via asources.tomltoml file. This change was necessary because we needed to introduce a map of tx source url -> consensus client url (so that when a new block is fetched from the tx source url, if a new block signer is detected, we can contact the node that created the block and get itsVerificationReport). Passing this map via command line was gnarly, so this moved to a configuration file (similar tonetwork.tomlinmc-consensus-service)WatcherDBnow includes additional databases (inside the LMDB file) for storingVerificationReports. The reports are indexed by (block signer public key, tx src url).Future Work
Unit tests forSee VerificationReportsCollector happy flow unit test eranrund/mobilecoin#16VerificationReportsCollector.mc-watcherstore all blocks fetched so that if the S3 sources ever goes away, the block data remains available locally.