[JIT-1661] Faster Autosnapshot #406

esemeniuc · 2023-10-10T15:21:26Z

Notes:

Fetches accounts in batches of 100
Retry fetching accounts
Avoid panic when, forcing account refetch
Avoid memory bloat with new blockhashes creating duplicate signatures
Log datapoints
Balance check delayed (avoid 2x balance requirement)

tip-distributor/README.md

tip-distributor/src/claim_mev_workflow.rs

tip-distributor/Cargo.toml

tip-distributor/src/bin/claim-mev-tips.rs

buffalu · 2023-10-12T19:10:48Z

tip-distributor/src/bin/claim-mev-tips.rs

+                args.merkle_trees_path.to_string_lossy(),
+                String
+            ),
+            ("latency_us", start.elapsed().as_micros(), i64),


nit: elapsed_us

zano usually suggests latency_us, hence been using that

latency seems like wrong word here, but just a nit

i've evolved from using latency for non-networking related measurements, that version of me is no longer present :)

buffalu · 2023-10-12T19:10:54Z

tip-distributor/src/bin/claim-mev-tips.rs

+                args.merkle_trees_path.to_string_lossy(),
+                String
+            ),
+            ("latency_us", start.elapsed().as_micros(), i64),


nit: elapsed_us

buffalu · 2023-10-12T19:45:02Z

tip-distributor/src/lib.rs

+            .expect("fetch latest blockhash"),
+    ));
+    let mut rng = rand::thread_rng();
+    transactions.shuffle(&mut rng); // shuffle to avoid sending same txns as other claim-tip processes


comment somewhat misleading, will still send the same? and have potential for failed txs?

shuffle is mainly if we run both warehouse nodes, would make them less likely to race for the same txns.

Still has potential for failed txns, but we refetch all accounts to see if they're claimed every couple mins to avoid duplicate work.

buffalu · 2023-10-12T19:45:57Z

tip-distributor/src/lib.rs

+    mut txn: Transaction,
+) -> (Transaction, solana_rpc_client_api::client_error::Result<()>) {
+    txn.sign(&[signer], blockhash); // just in time signing
+    let res = match rpc_client.send_and_confirm_transaction(&txn).await {


how long does this wait to confirm?

should try every 0.5 secs: https://github.com/jito-labs/jito-solana/blob/f6b2cc843b675653ecb9061d326e4e41b5b0dda1/rpc-client/src/nonblocking/rpc_client.rs#L708

it looks like max amount of time would be blockhash expiry?

yea looks like that

tip-distributor/src/lib.rs

buffalu · 2023-10-12T20:27:07Z

tip-distributor/src/lib.rs

+pub async fn sign_and_send_transactions_with_retries_multi_rpc(
+    signer: &Arc<Keypair>,
+    blockhash_rpc_client: &Arc<RpcClient>,
+    rpc_clients: &Arc<Vec<Arc<RpcClient>>>,


should make sure we used confirmed or finalized here. finalized will make it much slow though

if we used processed, might end up false confirming a block that gets forked out

they all use confirmed here:
https://github.com/jito-labs/jito-solana/blob/fda99db3060d4e2ac9dbb558f8944ac5ffa0c01a/tip-distributor/src/claim_mev_workflow.rs#L93-L102

buffalu · 2023-10-12T20:28:32Z

tip-distributor/src/lib.rs

+    mut txn: Transaction,
+) -> (Transaction, solana_rpc_client_api::client_error::Result<()>) {
+    txn.sign(&[signer], blockhash); // just in time signing
+    let res = match rpc_client.send_and_confirm_transaction(&txn).await {


it looks like max amount of time would be blockhash expiry?

tip-distributor/src/bin/claim-mev-tips.rs

buffalu · 2023-11-08T11:42:34Z

tip-distributor/Cargo.toml

@@ -46,7 +51,3 @@ path = "src/bin/merkle-root-uploader.rs"
 [[bin]]
 name = "solana-claim-mev-tips"
 path = "src/bin/claim-mev-tips.rs"
-


Why remove this?

merged into same binary

Use cluster info functions for tpu (#345) (#347) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (Backport #374) (#376) Updates scripts for easy local setup. (#383) Backports sim_bundle improvements (#407) backports clone derivation 416 (#417) Backport #419: add upsert to AccountOverrides (#420) backports #430 v1.16: update jito-programs sha (#431) [JIT-1661] Faster Autosnapshot (#406) Fix Buildkite warnings (#437) Backport #446 to v1.16 (#447) backport 428, runtime plugin (#429) v1.16: Backport #449 (#450)

Use cluster info functions for tpu (#345) (#347) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (Backport #374) (#376) Updates scripts for easy local setup. (#383) Backports sim_bundle improvements (#407) backports clone derivation 416 (#417) Backport #419: add upsert to AccountOverrides (#420) backports #430 v1.16: update jito-programs sha (#431) [JIT-1661] Faster Autosnapshot (#406) Fix Buildkite warnings (#437) Backport #446 to v1.16 (#447) backport 428, runtime plugin (#429) v1.16: Backport #449 (#450) [JIT-1713] Fix bundle's blockspace preallocation (Backport to 1.16) (#498) Backport #500 to v1.16 (#506)

Use cluster info functions for tpu (#345) (#347) Use git rev-parse for git sha Remove blacklisted tx from message_hash_to_transaction (Backport #374) (#376) Updates scripts for easy local setup. (#383) Backports sim_bundle improvements (#407) backports clone derivation 416 (#417) Backport #419: add upsert to AccountOverrides (#420) backports #430 v1.16: update jito-programs sha (#431) [JIT-1661] Faster Autosnapshot (#406) Fix Buildkite warnings (#437) Backport #446 to v1.16 (#447) backport 428, runtime plugin (#429) v1.16: Backport #449 (#450) [JIT-1713] Fix bundle's blockspace preallocation (Backport to 1.16) (#498) Backport #500 to v1.16 (#506) backports #520: add priority fees to mev claim (#524) Backport #548 (#549)

esemeniuc requested review from buffalu, jedleggett and segfaultdoc October 10, 2023 15:21

segfaultdoc reviewed Oct 11, 2023

View reviewed changes

tip-distributor/README.md Outdated Show resolved Hide resolved

segfaultdoc reviewed Oct 11, 2023

View reviewed changes

tip-distributor/src/claim_mev_workflow.rs Outdated Show resolved Hide resolved

segfaultdoc reviewed Oct 11, 2023

View reviewed changes

tip-distributor/src/claim_mev_workflow.rs Show resolved Hide resolved

buffalu force-pushed the v1.16 branch 2 times, most recently from 75652a8 to f6b2cc8 Compare October 11, 2023 21:20

esemeniuc force-pushed the eric/tip-dist branch from 3e5dcb0 to fda99db Compare October 12, 2023 16:15

buffalu reviewed Oct 12, 2023

View reviewed changes

tip-distributor/Cargo.toml Outdated Show resolved Hide resolved

buffalu reviewed Oct 12, 2023

View reviewed changes

segfaultdoc approved these changes Oct 12, 2023

View reviewed changes

buffalu reviewed Oct 12, 2023

View reviewed changes

buffalu force-pushed the v1.16 branch from 56955bf to 081bdbf Compare October 15, 2023 19:33

segfaultdoc self-requested a review October 24, 2023 15:09

segfaultdoc reviewed Oct 24, 2023

View reviewed changes

tip-distributor/src/bin/claim-mev-tips.rs Outdated Show resolved Hide resolved

buffalu force-pushed the v1.16 branch 2 times, most recently from 4696aa6 to 70c3f57 Compare October 27, 2023 17:19

add files

2a4b508

esemeniuc force-pushed the eric/tip-dist branch from dd3cf7e to 2a4b508 Compare November 8, 2023 10:29

esemeniuc changed the title ~~Faster Autosnapshot~~ [JIT-1661] Faster Autosnapshot Nov 8, 2023

esemeniuc merged commit 2a8d046 into v1.16 Nov 8, 2023

esemeniuc deleted the eric/tip-dist branch November 8, 2023 11:23

esemeniuc restored the eric/tip-dist branch November 8, 2023 11:24

esemeniuc deleted the eric/tip-dist branch November 8, 2023 11:24

esemeniuc mentioned this pull request Nov 8, 2023

[JIT-1661] Faster Autosnapshot #436

Merged

buffalu reviewed Nov 8, 2023

View reviewed changes

esemeniuc mentioned this pull request Nov 8, 2023

Fix Buildkite warnings #437

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JIT-1661] Faster Autosnapshot #406

[JIT-1661] Faster Autosnapshot #406

esemeniuc commented Oct 10, 2023 •

edited

Loading

buffalu Oct 12, 2023

esemeniuc Oct 12, 2023

buffalu Oct 12, 2023

segfaultdoc Oct 24, 2023

buffalu Oct 12, 2023

buffalu Oct 12, 2023

esemeniuc Oct 12, 2023

esemeniuc Oct 12, 2023

buffalu Oct 12, 2023

esemeniuc Oct 12, 2023

buffalu Oct 12, 2023

esemeniuc Oct 12, 2023

buffalu Oct 12, 2023

esemeniuc Oct 12, 2023

buffalu Oct 12, 2023

buffalu Nov 8, 2023

esemeniuc Nov 8, 2023

[JIT-1661] Faster Autosnapshot #406

[JIT-1661] Faster Autosnapshot #406

Conversation

esemeniuc commented Oct 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

esemeniuc commented Oct 10, 2023 •

edited

Loading