chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version #5941

kevaundray · 2024-06-17T18:40:54Z

Issue Addressed

This relies on #5938

Below was copied from Dapplion:

Description: Replace c-kzg peerdas crypto with new lib peerdas-kzg
Rationale: Do not touch mainnet paths, and use this lib only for das codepaths
Justification: Eventually this lib should become canonical, and we can add a switch later when ckzg peerdas merges to main

Proposed Changes

Please list or describe the changes introduced by this PR.

Additional Info

Please provide any additional information. For example, future considerations
or information useful for reviewers.

- update `ssz_cell_to_crypto_cell` - update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec`

- update rest of code to handle the new crypto `Cell` type - update test case code to no longer use the Box type

kevaundray · 2024-06-17T18:42:05Z

beacon_node/beacon_chain/src/kzg_utils.rs

@@ -11,8 +11,11 @@ fn ssz_blob_to_crypto_blob<E: EthSpec>(blob: &Blob<E>) -> Result<KzgBlob, KzgErr

 /// Converts a cell ssz List object to an array to be used with the kzg
 /// crypto library.
-fn ssz_cell_to_crypto_cell<E: EthSpec>(cell: &Cell<E>) -> Result<KzgCell, KzgError> {
-    KzgCell::from_bytes(cell.as_ref()).map_err(Into::into)
+fn ssz_cell_to_crypto_cell<E: EthSpec>(cell: &Cell<E>) -> Result<KzgCellRef, KzgError> {


Noting that this method is duplicated in data_column_sidecar.rs

Cool, we can probably remove this one

kevaundray · 2024-06-17T18:55:11Z

crypto/kzg/src/lib.rs

        &self,
        cell_ids: &[u64],
-        cells: &[Cell],
-    ) -> Result<Box<[Cell; c_kzg::CELLS_PER_EXT_BLOB]>, Error> {


I don't believe the way Box was being used before is guaranteed to always allocate on the heap, there is a tracking issue here for it: rust-lang/rust#27779 (comment)

Its unclear to me in what cases the compiler will be smart enough to allocate on the heap directly, so this seems like it may end up with sporadic stackoverflows

The crypto lib tries to mitigate this by taking Cell as Vec, then Boxing it, and turning the Boxed Vec into a Boxed [u8;N]

This looks to be reliable, though I can see an argument for returning Vec -- you just lose type safety

testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof.rs

kevaundray · 2024-06-17T19:01:07Z

crypto/kzg/src/lib.rs

+        let (cells, proofs) = self
+            .context
+            .prover_ctx()
+            .recover_cells_and_proofs(cell_ids.to_vec(), cells.to_vec(), vec![])


We can improve on the crypto API here, and take &[Cell] instead of Vec, same for other parameters.

proofs is left as an empty parameter because the underlying library does not use it, it just recomputes every proof each time

crypto/kzg/src/lib.rs

- This version includes a new method `PeerDasContext::with_num_threads`

… set the parameters in `new_from_trusted_setup`

kevaundray · 2024-06-17T19:45:19Z

crypto/kzg/src/lib.rs

+        // Set the number of threads to be used
+        //
+        // we set it to 1 to match the c-kzg performance
+        const NUM_THREADS : usize = 1;


This can be set to something sensible in another PR

kevaundray · 2024-06-17T19:46:45Z

crypto/kzg/src/lib.rs

-                // Enable precomputed table for 8 bits, with 96MB of memory overhead per process
-                // Ref: https://notes.ethereum.org/@jtraglia/windowed_multiplications
-                8,


This parameter does not exist on the c-kzg 4844 auditted branch -- since I've reverted c-kzg back to v1, it has been removed

kevaundray · 2024-06-17T19:52:22Z

crypto/kzg/src/lib.rs

@@ -13,13 +13,21 @@ pub use c_kzg::{
    Blob, Bytes32, Bytes48, KzgSettings, BYTES_PER_BLOB, BYTES_PER_COMMITMENT,
    BYTES_PER_FIELD_ELEMENT, BYTES_PER_PROOF, FIELD_ELEMENTS_PER_BLOB,
 };
-pub use c_kzg::{Cell, CELLS_PER_EXT_BLOB};


Note: the only type which I've changed/removed is the Cell type -- The Blob types etc that are needed for 4844 are left unchanged along with their codepaths.

We convert from c-kzg Blob type to PeerDASKZG blob type inside of this file.

This is just so that we can be sure that the old c-kzg code has been left untouched.

Cargo.toml

crypto/kzg/src/lib.rs

jimmygchen · 2024-06-19T23:47:19Z

Thanks @kevaundray! Are there any benchmarks for the peerdas-kzg crate?

kevaundray · 2024-06-20T22:29:34Z

Thanks @kevaundray! Are there any benchmarks for the peerdas-kzg crate?

Yep, I added some a few hours ago -- you can cd eip7594 and cargo bench -- For a single thread, it should have around the same performance as c-kzg -- for computing cells and proofs, I get ~2x when I set NUM_THREADS to 8. I expect this number to be driven down further when more components are parallelized

Co-authored-by: Jimmy Chen <[email protected]>

kevaundray · 2024-06-27T10:37:07Z

@jimmygchen I've ran the ef-tests locally and make lint -- will leave local devnet testing and other blockers in your hand

jimmygchen · 2024-06-28T12:12:36Z

Sure, I'll do the testing, thanks a lot for this ☺️

Sorry about the delay in responding! and just to answer your questions:

Is there a way to check if there are performance issues before merging?

Yep we do have some kzg metrics exposed via the {host}:{metrics_port}/metrics endpoint, and may be useful to compare them:

> curl -s $(kurtosis port print local-testnet cl-2-lighthouse-geth metrics)/metrics | grep -E "HELP.+column"
# HELP beacon_blobs_column_sidecar_gossip_verification_seconds Full runtime of data column sidecars gossip verification
# HELP beacon_blobs_column_sidecar_processing_requests_total Count of all data column sidecars submitted for processing
# HELP beacon_blobs_column_sidecar_processing_successes_total Number of data column sidecars verified for gossip
# HELP beacon_data_column_gossip_propagation_verification_delay_time Duration between when the data column sidecar is received over gossip and when it is verified for propagation.
# HELP beacon_data_column_gossip_slot_start_delay_time Duration between when the data column sidecar is received over gossip and the start of the slot it belongs to.
# HELP beacon_data_column_last_delay Keeps track of the last data column sidecar's delay from the start of the slot
# HELP beacon_processor_gossip_data_column_queue_total Count of data column sidecars from gossip waiting to be verified.
# HELP beacon_processor_gossip_data_column_verified_total Total number of gossip data column sidecar verified for propagation.
# HELP beacon_processor_rpc_custody_column_queue_total Count of custody columns from the rpc waiting to be imported.
# HELP beacon_processor_rpc_verify_data_column_queue_total Count of data columns from the rpc waiting to be verified.
# HELP data_availability_reconstructed_columns_total Total count of reconstructed columns
# HELP data_availability_reconstruction_time_seconds Time taken to reconstruct columns
# HELP data_column_sidecar_computation_seconds Time taken to compute data column sidecar, including cells, proofs and inclusion proof
# HELP data_column_sidecar_inclusion_proof_verification_seconds Time taken to verify data_column sidecar inclusion proof
# HELP kzg_verification_data_column_single_seconds Runtime of single data column kzg verification

Were you looking for something in particular wrt to testing it locally?

I think the main thing to look for when testing peerdas is making sure the network continue to produce blocks and nodes are in sync with each other - this means columns are being produced and propagated and likely works end-to-end. And if we want to make sure it interops with c-kzg, it might be worth running the network-params-das-interop.yaml one (with prysm and teku).

kevaundray · 2024-06-28T13:14:21Z

Sure, I'll do the testing, thanks a lot for this ☺️

Sorry about the delay in responding! and just to answer your questions:

Is there a way to check if there are performance issues before merging?

Yep we do have some kzg metrics exposed via the {host}:{metrics_port}/metrics endpoint, and may be useful to compare them:


> curl -s $(kurtosis port print local-testnet cl-2-lighthouse-geth metrics)/metrics | grep -E "HELP.+column"

# HELP beacon_blobs_column_sidecar_gossip_verification_seconds Full runtime of data column sidecars gossip verification

# HELP beacon_blobs_column_sidecar_processing_requests_total Count of all data column sidecars submitted for processing

# HELP beacon_blobs_column_sidecar_processing_successes_total Number of data column sidecars verified for gossip

# HELP beacon_data_column_gossip_propagation_verification_delay_time Duration between when the data column sidecar is received over gossip and when it is verified for propagation.

# HELP beacon_data_column_gossip_slot_start_delay_time Duration between when the data column sidecar is received over gossip and the start of the slot it belongs to.

# HELP beacon_data_column_last_delay Keeps track of the last data column sidecar's delay from the start of the slot

# HELP beacon_processor_gossip_data_column_queue_total Count of data column sidecars from gossip waiting to be verified.

# HELP beacon_processor_gossip_data_column_verified_total Total number of gossip data column sidecar verified for propagation.

# HELP beacon_processor_rpc_custody_column_queue_total Count of custody columns from the rpc waiting to be imported.

# HELP beacon_processor_rpc_verify_data_column_queue_total Count of data columns from the rpc waiting to be verified.

# HELP data_availability_reconstructed_columns_total Total count of reconstructed columns

# HELP data_availability_reconstruction_time_seconds Time taken to reconstruct columns

# HELP data_column_sidecar_computation_seconds Time taken to compute data column sidecar, including cells, proofs and inclusion proof

# HELP data_column_sidecar_inclusion_proof_verification_seconds Time taken to verify data_column sidecar inclusion proof

# HELP kzg_verification_data_column_single_seconds Runtime of single data column kzg verification

Were you looking for something in particular wrt to testing it locally?

I think the main thing to look for when testing peerdas is making sure the network continue to produce blocks and nodes are in sync with each other - this means columns are being produced and propagated and likely works end-to-end. And if we want to make sure it interops with c-kzg, it might be worth running the network-params-das-interop.yaml one (with prysm and teku).

No worries! I can see you have your hands full :)

I'm working on integrations for a few other repos, so I'll probably have to circle back to collecting information next week on local devnet. This information is useful in general so if you get to it before me, feel free to post it here!

jimmygchen · 2024-07-07T23:56:14Z

beacon_node/beacon_chain/src/kzg_utils.rs

+    let cell_bytes: &[u8] = cell.as_ref();
+    Ok(cell_bytes
+        .try_into()
+        .expect("expected cell to have size {BYTES_PER_CELL}. This should be guaranteed by the `FixedVector type"))


Suggested change

.expect("expected cell to have size {BYTES_PER_CELL}. This should be guaranteed by the `FixedVector type"))

.expect("expected cell to have size {BYTES_PER_CELL}. This should be guaranteed by the `FixedVector` type"))

jimmygchen · 2024-07-08T08:02:40Z

Hi @kevaundray,

Apologies on the delay getting to this. I was out the last couple of days.

I did a quick round of testing locally and it seems to be fine with a Lighthouse only network, but when I ran it with Prysm and Teku, the network forked pretty quickly.

I'll look into it again tomorrow, maybe I'm missing something in my network config.

kevaundray · 2024-07-08T08:20:13Z

Hi @kevaundray,

Apologies on the delay getting to this. I was out the last couple of days.

I did a quick round of testing locally and it seems to be fine with a Lighthouse only network, but when I ran it with Prysm and Teku, the network forked pretty quickly.

I'll look into it again tomorrow, maybe I'm missing something in my network config.

No worries :)

I'm guessing the DAS branch without this change is okay?

jimmygchen · 2024-07-08T09:07:00Z

I’m not entirely sure tbh, didn’t get too much time on this today. I didn’t see any KZG errors so I suspect it may be related to mismatch das network defaults for different clients and we may have to override them in the Kurtosis config, I’ll try again tomorrow.

jimmygchen · 2024-07-09T06:11:29Z

I've tested again, looking pretty good after I added network config overrides, so I guess clients currently have different default values:

participants:
  - cl_type: prysm
    cl_image: ethpandaops/prysm-beacon-chain:peerDAS

  - cl_type: lighthouse
    cl_extra_params: [
      --subscribe-all-data-column-subnets,
    ]
    cl_image: lighthouse:local

  - cl_type: lighthouse
    cl_image: lighthouse:local

  - cl_type: teku
    cl_image: ethpandaops/teku:nashatyrev-das

#  - cl_type: nimbus
#    cl_image: ethpandaops/nimbus-eth2:kzgpeerdas
#
#  - cl_type: grandine
#    cl_image: ethpandaops/grandine:das
#
#   - cl_type: lodestar
#     cl_image: ethpandaops/lodestar:peerDAS
network_params:
  eip7594_fork_epoch: 0
  eip7594_fork_version: "0x50000038"
  data_column_sidecar_subnset_count: 64
  samples_per_slot: 16
  custody_requirement: 4
  target_number_of_peers: 70
snooper_enabled: false
global_log_level: debug
additional_services:
  - dora
  - goomy_blob

I'll post some metrics comparison soon

jimmygchen · 2024-07-09T08:03:42Z

Metrics running this branch, looks pretty good and comparable to das branch.

This branch (over 1 h):

das branch (over 5 m):

jimmygchen · 2024-07-09T08:08:23Z

@dapplion I've tested the branch and it works pretty well!

I've attached some metrics in the above comment. I've ran out of time to review the new changes, but feel free to merge if you're happy with it, otherwise I'll re-review it when I have a chance.

Thanks a lot for this @kevaundray ! 🙏

…ell kzg verification (sigp#5701, sigp#5941, sigp#6118, sigp#6179) Co-authored-by: kevaundray <[email protected]>

) * Add peerdas KZG library and use it for data column construction and cell kzg verification (#5701, #5941, #6118, #6179) Co-authored-by: kevaundray <[email protected]> * Update `rust_eth_kzg` crate to published version. * Update kzg metrics buckets. * Merge branch 'unstable' into peerdas-kzg * Update KZG version to fix windows mem allocation. * Refactor common logic from build sidecar and reconstruction. Remove unnecessary `needless_lifetimes`. Co-authored-by: realbigsean <[email protected]> * Copy existing trusted setup into `PeerDASTrustedSetup` for consistency and maintain `--trusted-setup` functionality. * Merge branch 'unstable' into peerdas-kzg * Merge branch 'peerdas-kzg' of github.com:jimmygchen/lighthouse into peerdas-kzg * Merge branch 'unstable' into peerdas-kzg * Merge branch 'unstable' into peerdas-kzg * Load PeerDAS KZG only if PeerDAS is enabled.

@jimmygchen

* 1D PeerDAS prototype: Data format and Distribution (#5050) * Build and publish column sidecars. Add stubs for gossip. * Add blob column subnets * Add `BlobColumnSubnetId` and initial compute subnet logic. * Subscribe to blob column subnets. * Introduce `BLOB_COLUMN_SUBNET_COUNT` based on DAS configuration parameter changes. * Fix column sidecar type to use `VariableList` for data. * Fix lint errors. * Update types and naming to latest consensus-spec #3574. * Fix test and some cleanups. * Merge branch 'unstable' into das * Merge branch 'unstable' into das * Merge branch 'unstable' into das # Conflicts: # consensus/types/src/chain_spec.rs * Add `DataColumnSidecarsByRoot ` req/resp protocol (#5196) * Add stub for `DataColumnsByRoot` * Add basic implementation of serving RPC data column from DA checker. * Store data columns in early attester cache and blobs db. * Apply suggestions from code review Co-authored-by: Eitan Seri-Levi <[email protected]> Co-authored-by: Jacob Kaufmann <[email protected]> * Fix build. * Store `DataColumnInfo` in database and various cleanups. * Update `DataColumnSidecar` ssz max size and remove panic code. --------- Co-authored-by: Eitan Seri-Levi <[email protected]> Co-authored-by: Jacob Kaufmann <[email protected]> * feat: add DAS KZG in data col construction (#5210) * feat: add DAS KZG in data col construction * refactor data col sidecar construction * refactor: add data cols to GossipVerifiedBlockContents * Disable windows tests for `das` branch. (c-kzg doesn't build on windows) * Formatting and lint changes only. * refactor: remove iters in construction of data cols * Update vec capacity and error handling. * Add `data_column_sidecar_computation_seconds` metric. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/lighthouse_network/src/types/topics.rs * fix: update data col subnet count from 64 to 32 (#5413) * feat: add peerdas custody field to ENR (#5409) * feat: add peerdas custody field to ENR * add hash prefix step in subnet computation * refactor test and fix possible u64 overflow * default to min custody value if not present in ENR * Merge branch 'unstable' into das * Merge branch 'unstable' into das-unstable-merge-0415 # Conflicts: # Cargo.lock # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/sync/block_lookups/tests.rs # crypto/kzg/Cargo.toml * Merge remote-tracking branch 'sigp/unstable' into das * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflicts. * Send custody data column to `DataAvailabilityChecker` for determining block importability (#5570) * Only import custody data columns after publishing a block. * Add `subscribe-all-data-column-subnets` and pass custody column count to `availability_cache`. * Add custody requirement checks to `availability_cache`. * Fix config not being passed to DAChecker and add more logging. * Introduce `peer_das_epoch` and make blobs and columns mutually exclusive. * Add DA filter for PeerDAS. * Fix data availability check and use test_logger in tests. * Fix subscribe to all data column subnets not working correctly. * Fix tests. * Only publish column sidecars if PeerDAS is activated. Add `PEER_DAS_EPOCH` chain spec serialization. * Remove unused data column index in `OverflowKey`. * Fix column sidecars incorrectly produced when there are no blobs. * Re-instate index to `OverflowKey::DataColumn` and downgrade noisy debug log to `trace`. * DAS sampling on sync (#5616) * Data availability sampling on sync * Address @jimmygchen review * Trigger sampling * Address some review comments and only send `SamplingBlock` sync message after PEER_DAS_EPOCH. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/block_verification.rs # beacon_node/http_api/src/publish_blocks.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/types/pubsub.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/store/src/hot_cold_store.rs # consensus/types/src/beacon_state.rs # consensus/types/src/chain_spec.rs # consensus/types/src/eth_spec.rs * Merge branch 'unstable' into das * Re-process early sampling requests (#5569) * Re-process early sampling requests # Conflicts: # beacon_node/beacon_processor/src/work_reprocessing_queue.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs * Update beacon_node/beacon_processor/src/work_reprocessing_queue.rs Co-authored-by: Jimmy Chen <[email protected]> * Add missing var * Beta compiler fixes and small typo fixes. * Remove duplicate method. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflict. * Add data columns by root to currently supported protocol list (#5678) * Add data columns by root to currently supported protocol list. * Add missing data column by roots handling. * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs * Fix simulator tests on `das` branch (#5731) * Bump genesis delay in sim tests as KZG setup takes longer for DAS. * Fix incorrect YAML spacing. * DataColumnByRange boilerplate (#5353) * add boilerplate * fmt * PeerDAS custody lookup sync (#5684) * Implement custody sync * Lint * Fix tests * Fix rebase issue * Add data column kzg verification and update `c-kzg`. (#5701) * Add data column kzg verification and update `c-kzg`. * Fix incorrect `Cell` size. * Add kzg verification on rpc blocks. * Add kzg verification on rpc data columns. * Rename `PEER_DAS_EPOCH` to `EIP7594_FORK_EPOCH` for client interop. (#5750) * Fetch custody columns in range sync (#5747) * Fetch custody columns in range sync * Clean up todos * Remove `BlobSidecar` construction and publish after PeerDAS activated (#5759) * Avoid building and publishing blob sidecars after PeerDAS. * Ignore gossip blobs with a slot greater than peer das activation epoch. * Only attempt to verify blob count and import blobs before PeerDAS. * #5684 review comments (#5748) * #5684 review comments. * Doc and message update only. * Fix incorrect condition when constructing `RpcBlock` with `DataColumn`s * Make sampling tests deterministic (#5775) * PeerDAS spec tests (#5772) * Add get_custody_columns spec tests. * Add kzg merkle proof spec tests. * Add SSZ spec tests. * Add remaining KZG tests * Load KZG only once per process, exclude electra tests and add missing SSZ tests. * Fix lint and missing changes. * Ignore macOS generated file. * Merge remote branch 'sigp/unstable' into das * Merge remote tracking branch 'origin/unstable' into das * Implement unconditional reconstruction for supernodes (#5781) * Implement unconditional reconstruction for supernodes * Move code into KzgVerifiedCustodyDataColumn * Remove expect * Add test * Thanks justin * Add withhold attack mode for interop (#5788) * Add withhold attack mode * Update readme * Drop added readmes * Undo styling changes * Add column gossip verification and handle unknown parent block (#5783) * Add column gossip verification and handle missing parent for columns. * Review PR * Fix rebase issue * more lint issues :) --------- Co-authored-by: dapplion <[email protected]> * Trigger sampling on sync events (#5776) * Trigger sampling on sync events * Update beacon_chain.rs * Fix tests * Fix tests * PeerDAS parameter changes for devnet-0 (#5779) * Update PeerDAS parameters to latest values. * Lint fix * Fix lint. * Update hardcoded subnet count to 64 (#5791) * Fix incorrect columns per subnet and config cleanup (#5792) * Tidy up PeerDAS preset and config values. * Fix broken config * Fix DAS branch CI (#5793) * Fix invalid syntax. * Update cli doc. Ignore get_custody_columns test temporarily. * Fix failing test and add verify inclusion test. * Undo accidentally removed code. * Only attempt reconstruct columns once. (#5794) * Re-enable precompute table for peerdas kzg (#5795) * Merge branch 'unstable' into das * Update subscription filter. (#5797) * Remove penalty for duplicate columns (expected due to reconstruction) (#5798) * Revert DAS config for interop testing. Optimise get_custody_columns function. (#5799) * Don't perform reconstruction for proposer node as it already has all the columns. (#5806) * Multithread compute_cells_and_proofs (#5805) * Multi-thread reconstruct data columns * Multi-thread path for block production * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/network_context.rs * Fix CI errors. * Move PeerDAS type-level config to configurable `ChainSpec` (#5828) * Move PeerDAS type level config to `ChainSpec`. * Fix tests * Misc custody lookup improvements (#5821) * Improve custody requests * Type DataColumnsByRootRequestId * Prioritize peers and load balance * Update tests * Address PR review * Merge branch 'unstable' into das * Rename deploy_block in network config (`das` branch) (#5852) * Rename deploy_block.txt to deposit_contract_block.txt * fmt --------- Co-authored-by: Pawan Dhananjay <[email protected]> * Merge branch 'unstable' into das * Fix CI and merge issues. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # lcli/src/main.rs * Store data columns individually in store and caches (#5890) * Store data columns individually in store and caches * Implement data column pruning * Merge branch 'unstable' into das # Conflicts: # Cargo.lock * Update reconstruction benches to newer criterion version. (#5949) * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml * chore: add `recover_cells_and_compute_proofs` method (#5938) * chore: add recover_cells_and_compute_proofs method * Introduce type alias `CellsAndKzgProofs` to address type complexity. --------- Co-authored-by: Jimmy Chen <[email protected]> * Update `csc` format in ENR and spec tests for devnet-1 (#5966) * Update `csc` format in ENR. * Add spec tests for `recover_cells_and_kzg_proofs`. * Add tests for ENR. * Fix failing tests. * Add protection against invalid csc value in ENR. * Fix lint * Fix csc encoding and decoding (#5997) * Fix data column rpc request not being sent due to incorrect limits set. (#6000) * Fix incorrect inbound request count causing rate limiting. (#6025) * Merge branch 'stable' into das # Conflicts: # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/block_sidecar_coupling.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # beacon_node/network/src/sync/network_context/requests.rs * Merge remote-tracking branch 'unstable' into das * Add kurtosis config for DAS testing (#5968) * Add kurtosis config for DAS testing. * Fix invalid yaml file * Update network parameter files. * chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version (#5941) * chore: add recover_cells_and_compute_proofs method * chore: add rust peerdas crypto library * chore: integrate peerdaskzg rust library into kzg crate * chore(multi): - update `ssz_cell_to_crypto_cell` - update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec` * chore(multi): - update rest of code to handle the new crypto `Cell` type - update test case code to no longer use the Box type * chore: cleanup of superfluous conversions * chore: revert c-kzg dependency back to v1 * chore: move dependency into correct order * chore: update rust dependency - This version includes a new method `PeerDasContext::with_num_threads` * chore: remove Default initialization of PeerDasContext and explicitly set the parameters in `new_from_trusted_setup` * chore: cleanup exports * chore: commit updated cargo.lock * Update Cargo.toml Co-authored-by: Jimmy Chen <[email protected]> * chore: rename dependency * chore: update peerdas lib - sets the blst version to 0.3 so that it matches whatever lighthouse is using. Although 0.3.12 is latest, lighthouse is pinned to 0.3.3 * chore: fix clippy lifetime - Rust doesn't allow you to elide the lifetime on type aliases * chore: cargo clippy fix * chore: cargo fmt * chore: update lib to add redundant checks (these will be removed in consensus-specs PR 3819) * chore: update dependency to ignore proofs * chore: update peerdas lib to latest * update lib * chore: remove empty proof parameter --------- Co-authored-by: Jimmy Chen <[email protected]> * Update PeerDAS interop testnet config (#6069) * Update interop testnet config. * Fix typo and remove target peers * Avoid retrying same sampling peer that previously failed. (#6084) * Various fixes to custody range sync (#6004) * Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests. * Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. * Add check to cover a case where batch is not processed while waiting for custody peers to become available. * Fix lint and logic error * Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`. * Add test for `get_custody_peers_for_column` * Revert epoch parameter refactor. * Fall back to default custody requiremnt if peer ENR is not present. * Add metrics and update code comment. * Add more debug logs. * Use subscribed peers on subnet before MetaDataV3 is implemented. Remove peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible. * Remove unused code and update docs. * Add comment * chore: update peerdas-kzg library (#6118) * chore: update peerDAS lib * chore: update library * chore: update library to version that include "init context" benchmarks and optional validation checks * chore: (can remove) -- Add benchmarks for init context * Prevent continuous searchers for low-peer networks (#6162) * Merge branch 'unstable' into das * Fix merge conflicts * Add cli flag to enable sampling and disable by default. (#6209) * chore: Use reference to an array representing a blob instead of an owned KzgBlob (#6179) * add KzgBlobRef type * modify code to use KzgBlobRef * clippy * Remove Deneb blob related changes to maintain compatibility with `c-kzg-4844`. --------- Co-authored-by: Jimmy Chen <[email protected]> * Store computed custody subnets in PeerDB and fix custody lookup test (#6218) * Fix failing custody lookup tests. * Store custody subnets in PeerDB, fix custody lookup test and refactor some methods. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/block_verification_types.rs # beacon_node/beacon_chain/src/builder.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/early_attester_cache.rs # beacon_node/beacon_chain/src/historical_blocks.rs # beacon_node/beacon_chain/tests/store_tests.rs # beacon_node/lighthouse_network/src/discovery/enr.rs # beacon_node/network/src/service.rs # beacon_node/src/cli.rs # beacon_node/store/src/hot_cold_store.rs # beacon_node/store/src/lib.rs # lcli/src/generate_bootnode_enr.rs * Fix CI failures after merge. * Batch sampling requests by peer (#6256) * Batch sampling requests by peer * Fix clippy errors * Fix tests * Add column_index to error message for ease of tracing * Remove outdated comment * Fix range sync never evaluating request as finished, causing it to get stuck. (#6276) * Merge branch 'unstable' into das-0821-merge # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/kzg_utils.rs # beacon_node/beacon_chain/src/metrics.rs # beacon_node/beacon_processor/src/lib.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/config.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/rpc/outbound.rs # beacon_node/lighthouse_network/src/rpc/rate_limiter.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/lighthouse_network/src/types/globals.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs # beacon_node/network/src/network_beacon_processor/sync_methods.rs # beacon_node/network/src/sync/block_lookups/common.rs # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # consensus/types/src/data_column_sidecar.rs # crypto/kzg/Cargo.toml # crypto/kzg/benches/benchmark.rs # crypto/kzg/src/lib.rs * Fix custody tests and load PeerDAS KZG instead. * Fix ef tests and bench compilation. * Fix failing sampling test. * Merge pull request #6287 from jimmygchen/das-0821-merge Merge `unstable` into `das` 20240821 * Remove get_block_import_status * Merge branch 'unstable' into das * Re-enable Windows release tests. * Address some review comments. * Address more review comments and cleanups. * Comment out peer DAS KZG EF tests for now * Address more review comments and fix build. * Merge branch 'das' of github.com:sigp/lighthouse into das * Unignore Electra tests * Fix metric name * Address some of Pawan's review comments * Merge remote-tracking branch 'origin/unstable' into das * Update PeerDAS network parameters for peerdas-devnet-2 (#6290) * update subnet count & custody req * das network params * update ef tests --------- Co-authored-by: Jimmy Chen <[email protected]>

…gp#6212) * Add peerdas KZG library and use it for data column construction and cell kzg verification (sigp#5701, sigp#5941, sigp#6118, sigp#6179) Co-authored-by: kevaundray <[email protected]> * Update `rust_eth_kzg` crate to published version. * Update kzg metrics buckets. * Merge branch 'unstable' into peerdas-kzg * Update KZG version to fix windows mem allocation. * Refactor common logic from build sidecar and reconstruction. Remove unnecessary `needless_lifetimes`. Co-authored-by: realbigsean <[email protected]> * Copy existing trusted setup into `PeerDASTrustedSetup` for consistency and maintain `--trusted-setup` functionality. * Merge branch 'unstable' into peerdas-kzg * Merge branch 'peerdas-kzg' of github.com:jimmygchen/lighthouse into peerdas-kzg * Merge branch 'unstable' into peerdas-kzg * Merge branch 'unstable' into peerdas-kzg * Load PeerDAS KZG only if PeerDAS is enabled.

@jimmygchen

* 1D PeerDAS prototype: Data format and Distribution (sigp#5050) * Build and publish column sidecars. Add stubs for gossip. * Add blob column subnets * Add `BlobColumnSubnetId` and initial compute subnet logic. * Subscribe to blob column subnets. * Introduce `BLOB_COLUMN_SUBNET_COUNT` based on DAS configuration parameter changes. * Fix column sidecar type to use `VariableList` for data. * Fix lint errors. * Update types and naming to latest consensus-spec sigp#3574. * Fix test and some cleanups. * Merge branch 'unstable' into das * Merge branch 'unstable' into das * Merge branch 'unstable' into das # Conflicts: # consensus/types/src/chain_spec.rs * Add `DataColumnSidecarsByRoot ` req/resp protocol (sigp#5196) * Add stub for `DataColumnsByRoot` * Add basic implementation of serving RPC data column from DA checker. * Store data columns in early attester cache and blobs db. * Apply suggestions from code review Co-authored-by: Eitan Seri-Levi <[email protected]> Co-authored-by: Jacob Kaufmann <[email protected]> * Fix build. * Store `DataColumnInfo` in database and various cleanups. * Update `DataColumnSidecar` ssz max size and remove panic code. --------- Co-authored-by: Eitan Seri-Levi <[email protected]> Co-authored-by: Jacob Kaufmann <[email protected]> * feat: add DAS KZG in data col construction (sigp#5210) * feat: add DAS KZG in data col construction * refactor data col sidecar construction * refactor: add data cols to GossipVerifiedBlockContents * Disable windows tests for `das` branch. (c-kzg doesn't build on windows) * Formatting and lint changes only. * refactor: remove iters in construction of data cols * Update vec capacity and error handling. * Add `data_column_sidecar_computation_seconds` metric. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/lighthouse_network/src/types/topics.rs * fix: update data col subnet count from 64 to 32 (sigp#5413) * feat: add peerdas custody field to ENR (sigp#5409) * feat: add peerdas custody field to ENR * add hash prefix step in subnet computation * refactor test and fix possible u64 overflow * default to min custody value if not present in ENR * Merge branch 'unstable' into das * Merge branch 'unstable' into das-unstable-merge-0415 # Conflicts: # Cargo.lock # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/sync/block_lookups/tests.rs # crypto/kzg/Cargo.toml * Merge remote-tracking branch 'sigp/unstable' into das * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflicts. * Send custody data column to `DataAvailabilityChecker` for determining block importability (sigp#5570) * Only import custody data columns after publishing a block. * Add `subscribe-all-data-column-subnets` and pass custody column count to `availability_cache`. * Add custody requirement checks to `availability_cache`. * Fix config not being passed to DAChecker and add more logging. * Introduce `peer_das_epoch` and make blobs and columns mutually exclusive. * Add DA filter for PeerDAS. * Fix data availability check and use test_logger in tests. * Fix subscribe to all data column subnets not working correctly. * Fix tests. * Only publish column sidecars if PeerDAS is activated. Add `PEER_DAS_EPOCH` chain spec serialization. * Remove unused data column index in `OverflowKey`. * Fix column sidecars incorrectly produced when there are no blobs. * Re-instate index to `OverflowKey::DataColumn` and downgrade noisy debug log to `trace`. * DAS sampling on sync (sigp#5616) * Data availability sampling on sync * Address @jimmygchen review * Trigger sampling * Address some review comments and only send `SamplingBlock` sync message after PEER_DAS_EPOCH. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/block_verification.rs # beacon_node/http_api/src/publish_blocks.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/types/pubsub.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/store/src/hot_cold_store.rs # consensus/types/src/beacon_state.rs # consensus/types/src/chain_spec.rs # consensus/types/src/eth_spec.rs * Merge branch 'unstable' into das * Re-process early sampling requests (sigp#5569) * Re-process early sampling requests # Conflicts: # beacon_node/beacon_processor/src/work_reprocessing_queue.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs * Update beacon_node/beacon_processor/src/work_reprocessing_queue.rs Co-authored-by: Jimmy Chen <[email protected]> * Add missing var * Beta compiler fixes and small typo fixes. * Remove duplicate method. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflict. * Add data columns by root to currently supported protocol list (sigp#5678) * Add data columns by root to currently supported protocol list. * Add missing data column by roots handling. * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs * Fix simulator tests on `das` branch (sigp#5731) * Bump genesis delay in sim tests as KZG setup takes longer for DAS. * Fix incorrect YAML spacing. * DataColumnByRange boilerplate (sigp#5353) * add boilerplate * fmt * PeerDAS custody lookup sync (sigp#5684) * Implement custody sync * Lint * Fix tests * Fix rebase issue * Add data column kzg verification and update `c-kzg`. (sigp#5701) * Add data column kzg verification and update `c-kzg`. * Fix incorrect `Cell` size. * Add kzg verification on rpc blocks. * Add kzg verification on rpc data columns. * Rename `PEER_DAS_EPOCH` to `EIP7594_FORK_EPOCH` for client interop. (sigp#5750) * Fetch custody columns in range sync (sigp#5747) * Fetch custody columns in range sync * Clean up todos * Remove `BlobSidecar` construction and publish after PeerDAS activated (sigp#5759) * Avoid building and publishing blob sidecars after PeerDAS. * Ignore gossip blobs with a slot greater than peer das activation epoch. * Only attempt to verify blob count and import blobs before PeerDAS. * sigp#5684 review comments (sigp#5748) * sigp#5684 review comments. * Doc and message update only. * Fix incorrect condition when constructing `RpcBlock` with `DataColumn`s * Make sampling tests deterministic (sigp#5775) * PeerDAS spec tests (sigp#5772) * Add get_custody_columns spec tests. * Add kzg merkle proof spec tests. * Add SSZ spec tests. * Add remaining KZG tests * Load KZG only once per process, exclude electra tests and add missing SSZ tests. * Fix lint and missing changes. * Ignore macOS generated file. * Merge remote branch 'sigp/unstable' into das * Merge remote tracking branch 'origin/unstable' into das * Implement unconditional reconstruction for supernodes (sigp#5781) * Implement unconditional reconstruction for supernodes * Move code into KzgVerifiedCustodyDataColumn * Remove expect * Add test * Thanks justin * Add withhold attack mode for interop (sigp#5788) * Add withhold attack mode * Update readme * Drop added readmes * Undo styling changes * Add column gossip verification and handle unknown parent block (sigp#5783) * Add column gossip verification and handle missing parent for columns. * Review PR * Fix rebase issue * more lint issues :) --------- Co-authored-by: dapplion <[email protected]> * Trigger sampling on sync events (sigp#5776) * Trigger sampling on sync events * Update beacon_chain.rs * Fix tests * Fix tests * PeerDAS parameter changes for devnet-0 (sigp#5779) * Update PeerDAS parameters to latest values. * Lint fix * Fix lint. * Update hardcoded subnet count to 64 (sigp#5791) * Fix incorrect columns per subnet and config cleanup (sigp#5792) * Tidy up PeerDAS preset and config values. * Fix broken config * Fix DAS branch CI (sigp#5793) * Fix invalid syntax. * Update cli doc. Ignore get_custody_columns test temporarily. * Fix failing test and add verify inclusion test. * Undo accidentally removed code. * Only attempt reconstruct columns once. (sigp#5794) * Re-enable precompute table for peerdas kzg (sigp#5795) * Merge branch 'unstable' into das * Update subscription filter. (sigp#5797) * Remove penalty for duplicate columns (expected due to reconstruction) (sigp#5798) * Revert DAS config for interop testing. Optimise get_custody_columns function. (sigp#5799) * Don't perform reconstruction for proposer node as it already has all the columns. (sigp#5806) * Multithread compute_cells_and_proofs (sigp#5805) * Multi-thread reconstruct data columns * Multi-thread path for block production * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/network_context.rs * Fix CI errors. * Move PeerDAS type-level config to configurable `ChainSpec` (sigp#5828) * Move PeerDAS type level config to `ChainSpec`. * Fix tests * Misc custody lookup improvements (sigp#5821) * Improve custody requests * Type DataColumnsByRootRequestId * Prioritize peers and load balance * Update tests * Address PR review * Merge branch 'unstable' into das * Rename deploy_block in network config (`das` branch) (sigp#5852) * Rename deploy_block.txt to deposit_contract_block.txt * fmt --------- Co-authored-by: Pawan Dhananjay <[email protected]> * Merge branch 'unstable' into das * Fix CI and merge issues. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # lcli/src/main.rs * Store data columns individually in store and caches (sigp#5890) * Store data columns individually in store and caches * Implement data column pruning * Merge branch 'unstable' into das # Conflicts: # Cargo.lock * Update reconstruction benches to newer criterion version. (sigp#5949) * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml * chore: add `recover_cells_and_compute_proofs` method (sigp#5938) * chore: add recover_cells_and_compute_proofs method * Introduce type alias `CellsAndKzgProofs` to address type complexity. --------- Co-authored-by: Jimmy Chen <[email protected]> * Update `csc` format in ENR and spec tests for devnet-1 (sigp#5966) * Update `csc` format in ENR. * Add spec tests for `recover_cells_and_kzg_proofs`. * Add tests for ENR. * Fix failing tests. * Add protection against invalid csc value in ENR. * Fix lint * Fix csc encoding and decoding (sigp#5997) * Fix data column rpc request not being sent due to incorrect limits set. (sigp#6000) * Fix incorrect inbound request count causing rate limiting. (sigp#6025) * Merge branch 'stable' into das # Conflicts: # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/block_sidecar_coupling.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # beacon_node/network/src/sync/network_context/requests.rs * Merge remote-tracking branch 'unstable' into das * Add kurtosis config for DAS testing (sigp#5968) * Add kurtosis config for DAS testing. * Fix invalid yaml file * Update network parameter files. * chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version (sigp#5941) * chore: add recover_cells_and_compute_proofs method * chore: add rust peerdas crypto library * chore: integrate peerdaskzg rust library into kzg crate * chore(multi): - update `ssz_cell_to_crypto_cell` - update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec` * chore(multi): - update rest of code to handle the new crypto `Cell` type - update test case code to no longer use the Box type * chore: cleanup of superfluous conversions * chore: revert c-kzg dependency back to v1 * chore: move dependency into correct order * chore: update rust dependency - This version includes a new method `PeerDasContext::with_num_threads` * chore: remove Default initialization of PeerDasContext and explicitly set the parameters in `new_from_trusted_setup` * chore: cleanup exports * chore: commit updated cargo.lock * Update Cargo.toml Co-authored-by: Jimmy Chen <[email protected]> * chore: rename dependency * chore: update peerdas lib - sets the blst version to 0.3 so that it matches whatever lighthouse is using. Although 0.3.12 is latest, lighthouse is pinned to 0.3.3 * chore: fix clippy lifetime - Rust doesn't allow you to elide the lifetime on type aliases * chore: cargo clippy fix * chore: cargo fmt * chore: update lib to add redundant checks (these will be removed in consensus-specs PR 3819) * chore: update dependency to ignore proofs * chore: update peerdas lib to latest * update lib * chore: remove empty proof parameter --------- Co-authored-by: Jimmy Chen <[email protected]> * Update PeerDAS interop testnet config (sigp#6069) * Update interop testnet config. * Fix typo and remove target peers * Avoid retrying same sampling peer that previously failed. (sigp#6084) * Various fixes to custody range sync (sigp#6004) * Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests. * Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. * Add check to cover a case where batch is not processed while waiting for custody peers to become available. * Fix lint and logic error * Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`. * Add test for `get_custody_peers_for_column` * Revert epoch parameter refactor. * Fall back to default custody requiremnt if peer ENR is not present. * Add metrics and update code comment. * Add more debug logs. * Use subscribed peers on subnet before MetaDataV3 is implemented. Remove peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible. * Remove unused code and update docs. * Add comment * chore: update peerdas-kzg library (sigp#6118) * chore: update peerDAS lib * chore: update library * chore: update library to version that include "init context" benchmarks and optional validation checks * chore: (can remove) -- Add benchmarks for init context * Prevent continuous searchers for low-peer networks (sigp#6162) * Merge branch 'unstable' into das * Fix merge conflicts * Add cli flag to enable sampling and disable by default. (sigp#6209) * chore: Use reference to an array representing a blob instead of an owned KzgBlob (sigp#6179) * add KzgBlobRef type * modify code to use KzgBlobRef * clippy * Remove Deneb blob related changes to maintain compatibility with `c-kzg-4844`. --------- Co-authored-by: Jimmy Chen <[email protected]> * Store computed custody subnets in PeerDB and fix custody lookup test (sigp#6218) * Fix failing custody lookup tests. * Store custody subnets in PeerDB, fix custody lookup test and refactor some methods. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/block_verification_types.rs # beacon_node/beacon_chain/src/builder.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/early_attester_cache.rs # beacon_node/beacon_chain/src/historical_blocks.rs # beacon_node/beacon_chain/tests/store_tests.rs # beacon_node/lighthouse_network/src/discovery/enr.rs # beacon_node/network/src/service.rs # beacon_node/src/cli.rs # beacon_node/store/src/hot_cold_store.rs # beacon_node/store/src/lib.rs # lcli/src/generate_bootnode_enr.rs * Fix CI failures after merge. * Batch sampling requests by peer (sigp#6256) * Batch sampling requests by peer * Fix clippy errors * Fix tests * Add column_index to error message for ease of tracing * Remove outdated comment * Fix range sync never evaluating request as finished, causing it to get stuck. (sigp#6276) * Merge branch 'unstable' into das-0821-merge # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/kzg_utils.rs # beacon_node/beacon_chain/src/metrics.rs # beacon_node/beacon_processor/src/lib.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/config.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/rpc/outbound.rs # beacon_node/lighthouse_network/src/rpc/rate_limiter.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/lighthouse_network/src/types/globals.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs # beacon_node/network/src/network_beacon_processor/sync_methods.rs # beacon_node/network/src/sync/block_lookups/common.rs # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # consensus/types/src/data_column_sidecar.rs # crypto/kzg/Cargo.toml # crypto/kzg/benches/benchmark.rs # crypto/kzg/src/lib.rs * Fix custody tests and load PeerDAS KZG instead. * Fix ef tests and bench compilation. * Fix failing sampling test. * Merge pull request sigp#6287 from jimmygchen/das-0821-merge Merge `unstable` into `das` 20240821 * Remove get_block_import_status * Merge branch 'unstable' into das * Re-enable Windows release tests. * Address some review comments. * Address more review comments and cleanups. * Comment out peer DAS KZG EF tests for now * Address more review comments and fix build. * Merge branch 'das' of github.com:sigp/lighthouse into das * Unignore Electra tests * Fix metric name * Address some of Pawan's review comments * Merge remote-tracking branch 'origin/unstable' into das * Update PeerDAS network parameters for peerdas-devnet-2 (sigp#6290) * update subnet count & custody req * das network params * update ef tests --------- Co-authored-by: Jimmy Chen <[email protected]>

…gp#6212) * Add peerdas KZG library and use it for data column construction and cell kzg verification (sigp#5701, sigp#5941, sigp#6118, sigp#6179) Co-authored-by: kevaundray <[email protected]> * Update `rust_eth_kzg` crate to published version. * Update kzg metrics buckets. * Merge branch 'unstable' into peerdas-kzg * Update KZG version to fix windows mem allocation. * Refactor common logic from build sidecar and reconstruction. Remove unnecessary `needless_lifetimes`. Co-authored-by: realbigsean <[email protected]> * Copy existing trusted setup into `PeerDASTrustedSetup` for consistency and maintain `--trusted-setup` functionality. * Merge branch 'unstable' into peerdas-kzg * Merge branch 'peerdas-kzg' of github.com:jimmygchen/lighthouse into peerdas-kzg * Merge branch 'unstable' into peerdas-kzg * Merge branch 'unstable' into peerdas-kzg * Load PeerDAS KZG only if PeerDAS is enabled.

@jimmygchen

* 1D PeerDAS prototype: Data format and Distribution (sigp#5050) * Build and publish column sidecars. Add stubs for gossip. * Add blob column subnets * Add `BlobColumnSubnetId` and initial compute subnet logic. * Subscribe to blob column subnets. * Introduce `BLOB_COLUMN_SUBNET_COUNT` based on DAS configuration parameter changes. * Fix column sidecar type to use `VariableList` for data. * Fix lint errors. * Update types and naming to latest consensus-spec sigp#3574. * Fix test and some cleanups. * Merge branch 'unstable' into das * Merge branch 'unstable' into das * Merge branch 'unstable' into das # Conflicts: # consensus/types/src/chain_spec.rs * Add `DataColumnSidecarsByRoot ` req/resp protocol (sigp#5196) * Add stub for `DataColumnsByRoot` * Add basic implementation of serving RPC data column from DA checker. * Store data columns in early attester cache and blobs db. * Apply suggestions from code review Co-authored-by: Eitan Seri-Levi <[email protected]> Co-authored-by: Jacob Kaufmann <[email protected]> * Fix build. * Store `DataColumnInfo` in database and various cleanups. * Update `DataColumnSidecar` ssz max size and remove panic code. --------- Co-authored-by: Eitan Seri-Levi <[email protected]> Co-authored-by: Jacob Kaufmann <[email protected]> * feat: add DAS KZG in data col construction (sigp#5210) * feat: add DAS KZG in data col construction * refactor data col sidecar construction * refactor: add data cols to GossipVerifiedBlockContents * Disable windows tests for `das` branch. (c-kzg doesn't build on windows) * Formatting and lint changes only. * refactor: remove iters in construction of data cols * Update vec capacity and error handling. * Add `data_column_sidecar_computation_seconds` metric. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/lighthouse_network/src/types/topics.rs * fix: update data col subnet count from 64 to 32 (sigp#5413) * feat: add peerdas custody field to ENR (sigp#5409) * feat: add peerdas custody field to ENR * add hash prefix step in subnet computation * refactor test and fix possible u64 overflow * default to min custody value if not present in ENR * Merge branch 'unstable' into das * Merge branch 'unstable' into das-unstable-merge-0415 # Conflicts: # Cargo.lock # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/availability_view.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_availability_checker/processing_cache.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/sync/block_lookups/tests.rs # crypto/kzg/Cargo.toml * Merge remote-tracking branch 'sigp/unstable' into das * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflicts. * Send custody data column to `DataAvailabilityChecker` for determining block importability (sigp#5570) * Only import custody data columns after publishing a block. * Add `subscribe-all-data-column-subnets` and pass custody column count to `availability_cache`. * Add custody requirement checks to `availability_cache`. * Fix config not being passed to DAChecker and add more logging. * Introduce `peer_das_epoch` and make blobs and columns mutually exclusive. * Add DA filter for PeerDAS. * Fix data availability check and use test_logger in tests. * Fix subscribe to all data column subnets not working correctly. * Fix tests. * Only publish column sidecars if PeerDAS is activated. Add `PEER_DAS_EPOCH` chain spec serialization. * Remove unused data column index in `OverflowKey`. * Fix column sidecars incorrectly produced when there are no blobs. * Re-instate index to `OverflowKey::DataColumn` and downgrade noisy debug log to `trace`. * DAS sampling on sync (sigp#5616) * Data availability sampling on sync * Address @jimmygchen review * Trigger sampling * Address some review comments and only send `SamplingBlock` sync message after PEER_DAS_EPOCH. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/block_verification.rs # beacon_node/http_api/src/publish_blocks.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/protocol.rs # beacon_node/lighthouse_network/src/types/pubsub.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/store/src/hot_cold_store.rs # consensus/types/src/beacon_state.rs # consensus/types/src/chain_spec.rs # consensus/types/src/eth_spec.rs * Merge branch 'unstable' into das * Re-process early sampling requests (sigp#5569) * Re-process early sampling requests # Conflicts: # beacon_node/beacon_processor/src/work_reprocessing_queue.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs * Update beacon_node/beacon_processor/src/work_reprocessing_queue.rs Co-authored-by: Jimmy Chen <[email protected]> * Add missing var * Beta compiler fixes and small typo fixes. * Remove duplicate method. --------- Co-authored-by: Jimmy Chen <[email protected]> * Merge remote-tracking branch 'sigp/unstable' into das * Fix merge conflict. * Add data columns by root to currently supported protocol list (sigp#5678) * Add data columns by root to currently supported protocol list. * Add missing data column by roots handling. * Merge branch 'unstable' into das # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs * Fix simulator tests on `das` branch (sigp#5731) * Bump genesis delay in sim tests as KZG setup takes longer for DAS. * Fix incorrect YAML spacing. * DataColumnByRange boilerplate (sigp#5353) * add boilerplate * fmt * PeerDAS custody lookup sync (sigp#5684) * Implement custody sync * Lint * Fix tests * Fix rebase issue * Add data column kzg verification and update `c-kzg`. (sigp#5701) * Add data column kzg verification and update `c-kzg`. * Fix incorrect `Cell` size. * Add kzg verification on rpc blocks. * Add kzg verification on rpc data columns. * Rename `PEER_DAS_EPOCH` to `EIP7594_FORK_EPOCH` for client interop. (sigp#5750) * Fetch custody columns in range sync (sigp#5747) * Fetch custody columns in range sync * Clean up todos * Remove `BlobSidecar` construction and publish after PeerDAS activated (sigp#5759) * Avoid building and publishing blob sidecars after PeerDAS. * Ignore gossip blobs with a slot greater than peer das activation epoch. * Only attempt to verify blob count and import blobs before PeerDAS. * sigp#5684 review comments (sigp#5748) * sigp#5684 review comments. * Doc and message update only. * Fix incorrect condition when constructing `RpcBlock` with `DataColumn`s * Make sampling tests deterministic (sigp#5775) * PeerDAS spec tests (sigp#5772) * Add get_custody_columns spec tests. * Add kzg merkle proof spec tests. * Add SSZ spec tests. * Add remaining KZG tests * Load KZG only once per process, exclude electra tests and add missing SSZ tests. * Fix lint and missing changes. * Ignore macOS generated file. * Merge remote branch 'sigp/unstable' into das * Merge remote tracking branch 'origin/unstable' into das * Implement unconditional reconstruction for supernodes (sigp#5781) * Implement unconditional reconstruction for supernodes * Move code into KzgVerifiedCustodyDataColumn * Remove expect * Add test * Thanks justin * Add withhold attack mode for interop (sigp#5788) * Add withhold attack mode * Update readme * Drop added readmes * Undo styling changes * Add column gossip verification and handle unknown parent block (sigp#5783) * Add column gossip verification and handle missing parent for columns. * Review PR * Fix rebase issue * more lint issues :) --------- Co-authored-by: dapplion <[email protected]> * Trigger sampling on sync events (sigp#5776) * Trigger sampling on sync events * Update beacon_chain.rs * Fix tests * Fix tests * PeerDAS parameter changes for devnet-0 (sigp#5779) * Update PeerDAS parameters to latest values. * Lint fix * Fix lint. * Update hardcoded subnet count to 64 (sigp#5791) * Fix incorrect columns per subnet and config cleanup (sigp#5792) * Tidy up PeerDAS preset and config values. * Fix broken config * Fix DAS branch CI (sigp#5793) * Fix invalid syntax. * Update cli doc. Ignore get_custody_columns test temporarily. * Fix failing test and add verify inclusion test. * Undo accidentally removed code. * Only attempt reconstruct columns once. (sigp#5794) * Re-enable precompute table for peerdas kzg (sigp#5795) * Merge branch 'unstable' into das * Update subscription filter. (sigp#5797) * Remove penalty for duplicate columns (expected due to reconstruction) (sigp#5798) * Revert DAS config for interop testing. Optimise get_custody_columns function. (sigp#5799) * Don't perform reconstruction for proposer node as it already has all the columns. (sigp#5806) * Multithread compute_cells_and_proofs (sigp#5805) * Multi-thread reconstruct data columns * Multi-thread path for block production * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/network_context.rs * Fix CI errors. * Move PeerDAS type-level config to configurable `ChainSpec` (sigp#5828) * Move PeerDAS type level config to `ChainSpec`. * Fix tests * Misc custody lookup improvements (sigp#5821) * Improve custody requests * Type DataColumnsByRootRequestId * Prioritize peers and load balance * Update tests * Address PR review * Merge branch 'unstable' into das * Rename deploy_block in network config (`das` branch) (sigp#5852) * Rename deploy_block.txt to deposit_contract_block.txt * fmt --------- Co-authored-by: Pawan Dhananjay <[email protected]> * Merge branch 'unstable' into das * Fix CI and merge issues. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # lcli/src/main.rs * Store data columns individually in store and caches (sigp#5890) * Store data columns individually in store and caches * Implement data column pruning * Merge branch 'unstable' into das # Conflicts: # Cargo.lock * Update reconstruction benches to newer criterion version. (sigp#5949) * Merge branch 'unstable' into das # Conflicts: # .github/workflows/test-suite.yml * chore: add `recover_cells_and_compute_proofs` method (sigp#5938) * chore: add recover_cells_and_compute_proofs method * Introduce type alias `CellsAndKzgProofs` to address type complexity. --------- Co-authored-by: Jimmy Chen <[email protected]> * Update `csc` format in ENR and spec tests for devnet-1 (sigp#5966) * Update `csc` format in ENR. * Add spec tests for `recover_cells_and_kzg_proofs`. * Add tests for ENR. * Fix failing tests. * Add protection against invalid csc value in ENR. * Fix lint * Fix csc encoding and decoding (sigp#5997) * Fix data column rpc request not being sent due to incorrect limits set. (sigp#6000) * Fix incorrect inbound request count causing rate limiting. (sigp#6025) * Merge branch 'stable' into das # Conflicts: # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/block_sidecar_coupling.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # beacon_node/network/src/sync/network_context/requests.rs * Merge remote-tracking branch 'unstable' into das * Add kurtosis config for DAS testing (sigp#5968) * Add kurtosis config for DAS testing. * Fix invalid yaml file * Update network parameter files. * chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version (sigp#5941) * chore: add recover_cells_and_compute_proofs method * chore: add rust peerdas crypto library * chore: integrate peerdaskzg rust library into kzg crate * chore(multi): - update `ssz_cell_to_crypto_cell` - update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec` * chore(multi): - update rest of code to handle the new crypto `Cell` type - update test case code to no longer use the Box type * chore: cleanup of superfluous conversions * chore: revert c-kzg dependency back to v1 * chore: move dependency into correct order * chore: update rust dependency - This version includes a new method `PeerDasContext::with_num_threads` * chore: remove Default initialization of PeerDasContext and explicitly set the parameters in `new_from_trusted_setup` * chore: cleanup exports * chore: commit updated cargo.lock * Update Cargo.toml Co-authored-by: Jimmy Chen <[email protected]> * chore: rename dependency * chore: update peerdas lib - sets the blst version to 0.3 so that it matches whatever lighthouse is using. Although 0.3.12 is latest, lighthouse is pinned to 0.3.3 * chore: fix clippy lifetime - Rust doesn't allow you to elide the lifetime on type aliases * chore: cargo clippy fix * chore: cargo fmt * chore: update lib to add redundant checks (these will be removed in consensus-specs PR 3819) * chore: update dependency to ignore proofs * chore: update peerdas lib to latest * update lib * chore: remove empty proof parameter --------- Co-authored-by: Jimmy Chen <[email protected]> * Update PeerDAS interop testnet config (sigp#6069) * Update interop testnet config. * Fix typo and remove target peers * Avoid retrying same sampling peer that previously failed. (sigp#6084) * Various fixes to custody range sync (sigp#6004) * Only start requesting batches when there are good peers across all custody columns to avoid spaming block requests. * Add custody peer check before mutating `BatchInfo` to avoid inconsistent state. * Add check to cover a case where batch is not processed while waiting for custody peers to become available. * Fix lint and logic error * Fix `good_peers_on_subnet` always returning false for `DataColumnSubnet`. * Add test for `get_custody_peers_for_column` * Revert epoch parameter refactor. * Fall back to default custody requiremnt if peer ENR is not present. * Add metrics and update code comment. * Add more debug logs. * Use subscribed peers on subnet before MetaDataV3 is implemented. Remove peer_id matching when injecting error because multiple peers are used for range requests. Use randomized custodial peer to avoid repeatedly sending requests to failing peers. Batch by range request where possible. * Remove unused code and update docs. * Add comment * chore: update peerdas-kzg library (sigp#6118) * chore: update peerDAS lib * chore: update library * chore: update library to version that include "init context" benchmarks and optional validation checks * chore: (can remove) -- Add benchmarks for init context * Prevent continuous searchers for low-peer networks (sigp#6162) * Merge branch 'unstable' into das * Fix merge conflicts * Add cli flag to enable sampling and disable by default. (sigp#6209) * chore: Use reference to an array representing a blob instead of an owned KzgBlob (sigp#6179) * add KzgBlobRef type * modify code to use KzgBlobRef * clippy * Remove Deneb blob related changes to maintain compatibility with `c-kzg-4844`. --------- Co-authored-by: Jimmy Chen <[email protected]> * Store computed custody subnets in PeerDB and fix custody lookup test (sigp#6218) * Fix failing custody lookup tests. * Store custody subnets in PeerDB, fix custody lookup test and refactor some methods. * Merge branch 'unstable' into das # Conflicts: # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/block_verification_types.rs # beacon_node/beacon_chain/src/builder.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/early_attester_cache.rs # beacon_node/beacon_chain/src/historical_blocks.rs # beacon_node/beacon_chain/tests/store_tests.rs # beacon_node/lighthouse_network/src/discovery/enr.rs # beacon_node/network/src/service.rs # beacon_node/src/cli.rs # beacon_node/store/src/hot_cold_store.rs # beacon_node/store/src/lib.rs # lcli/src/generate_bootnode_enr.rs * Fix CI failures after merge. * Batch sampling requests by peer (sigp#6256) * Batch sampling requests by peer * Fix clippy errors * Fix tests * Add column_index to error message for ease of tracing * Remove outdated comment * Fix range sync never evaluating request as finished, causing it to get stuck. (sigp#6276) * Merge branch 'unstable' into das-0821-merge # Conflicts: # Cargo.lock # Cargo.toml # beacon_node/beacon_chain/src/beacon_chain.rs # beacon_node/beacon_chain/src/data_availability_checker.rs # beacon_node/beacon_chain/src/data_availability_checker/overflow_lru_cache.rs # beacon_node/beacon_chain/src/data_column_verification.rs # beacon_node/beacon_chain/src/kzg_utils.rs # beacon_node/beacon_chain/src/metrics.rs # beacon_node/beacon_processor/src/lib.rs # beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs # beacon_node/lighthouse_network/src/rpc/config.rs # beacon_node/lighthouse_network/src/rpc/methods.rs # beacon_node/lighthouse_network/src/rpc/outbound.rs # beacon_node/lighthouse_network/src/rpc/rate_limiter.rs # beacon_node/lighthouse_network/src/service/api_types.rs # beacon_node/lighthouse_network/src/types/globals.rs # beacon_node/network/src/network_beacon_processor/mod.rs # beacon_node/network/src/network_beacon_processor/rpc_methods.rs # beacon_node/network/src/network_beacon_processor/sync_methods.rs # beacon_node/network/src/sync/block_lookups/common.rs # beacon_node/network/src/sync/block_lookups/mod.rs # beacon_node/network/src/sync/block_lookups/single_block_lookup.rs # beacon_node/network/src/sync/block_lookups/tests.rs # beacon_node/network/src/sync/manager.rs # beacon_node/network/src/sync/network_context.rs # consensus/types/src/data_column_sidecar.rs # crypto/kzg/Cargo.toml # crypto/kzg/benches/benchmark.rs # crypto/kzg/src/lib.rs * Fix custody tests and load PeerDAS KZG instead. * Fix ef tests and bench compilation. * Fix failing sampling test. * Merge pull request sigp#6287 from jimmygchen/das-0821-merge Merge `unstable` into `das` 20240821 * Remove get_block_import_status * Merge branch 'unstable' into das * Re-enable Windows release tests. * Address some review comments. * Address more review comments and cleanups. * Comment out peer DAS KZG EF tests for now * Address more review comments and fix build. * Merge branch 'das' of github.com:sigp/lighthouse into das * Unignore Electra tests * Fix metric name * Address some of Pawan's review comments * Merge remote-tracking branch 'origin/unstable' into das * Update PeerDAS network parameters for peerdas-devnet-2 (sigp#6290) * update subnet count & custody req * das network params * update ef tests --------- Co-authored-by: Jimmy Chen <[email protected]>

kevaundray added 6 commits June 17, 2024 17:49

chore: add recover_cells_and_compute_proofs method

8331583

chore: add rust peerdas crypto library

f84bbde

chore: integrate peerdaskzg rust library into kzg crate

2f0eefa

chore(multi):

2bec02c

- update `ssz_cell_to_crypto_cell` - update conversion from the crypto cell type to a Vec<u8>. Since the Rust library defines them as references to an array, the conversion is simply `to_vec`

chore(multi):

6f3c1b6

- update rest of code to handle the new crypto `Cell` type - update test case code to no longer use the Box type

chore: cleanup of superfluous conversions

d09a495

kevaundray commented Jun 17, 2024

View reviewed changes

testing/ef_tests/src/cases/kzg_verify_blob_kzg_proof.rs Show resolved Hide resolved

kevaundray commented Jun 17, 2024

View reviewed changes

crypto/kzg/src/lib.rs Outdated Show resolved Hide resolved

kevaundray added 4 commits June 17, 2024 20:22

chore: revert c-kzg dependency back to v1

d15a8ae

chore: move dependency into correct order

ed418a6

chore: update rust dependency

37552cb

- This version includes a new method `PeerDasContext::with_num_threads`

chore: remove Default initialization of PeerDasContext and explicitly…

5f142aa

… set the parameters in `new_from_trusted_setup`

kevaundray commented Jun 17, 2024

View reviewed changes

chore: cleanup exports

405e80e

kevaundray changed the title ~~chore: add rust peerdas kzg crypto library~~ chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version Jun 17, 2024

chong-he added the das Data Availability Sampling label Jun 18, 2024

jimmygchen self-assigned this Jun 18, 2024

jimmygchen added the ready-for-review The code is ready for review label Jun 18, 2024

dapplion reviewed Jun 18, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

crypto/kzg/src/lib.rs Show resolved Hide resolved

crypto/kzg/src/lib.rs Show resolved Hide resolved

kevaundray added 2 commits June 19, 2024 09:22

Merge branch 'das' into kw/add-peerdas-kzg-lib

629ba9f

chore: commit updated cargo.lock

fc34d59

Update Cargo.toml

fda359e

Co-authored-by: Jimmy Chen <[email protected]>

kevaundray requested a review from dapplion June 24, 2024 16:28

jimmygchen added the under-review A reviewer has only partially completed a review. label Jun 28, 2024

chore: update peerdas lib to latest

6440d78

kevaundray mentioned this pull request Jun 30, 2024

Tracking Issue: PeerDAS-KZG Integration (Consensus Layer) crate-crypto/rust-eth-kzg#42

Open

8 tasks

update lib

8570748

kevaundray force-pushed the kw/add-peerdas-kzg-lib branch from 24c9023 to 8570748 Compare July 4, 2024 16:45

kevaundray added 2 commits July 4, 2024 17:45

chore: remove empty proof parameter

2b2938a

Merge branch 'das' into kw/add-peerdas-kzg-lib

7d284bb

jimmygchen reviewed Jul 7, 2024

View reviewed changes

Merge branch 'das' into kw/add-peerdas-kzg-lib

d388f54

jimmygchen removed the under-review A reviewer has only partially completed a review. label Jul 9, 2024

dapplion merged commit 094ee60 into sigp:das Jul 9, 2024
24 of 26 checks passed

This was referenced Jul 16, 2024

Investigate PeerDAS KZG library stack overflow during block production #6105

Closed

Add option to select KZG library for PeerDAS #6107

Open

jimmygchen added a commit to jimmygchen/lighthouse that referenced this pull request Aug 1, 2024

Add peerdas KZG library and use it for data column construction and c…

88b6bd9

…ell kzg verification (sigp#5701, sigp#5941, sigp#6118, sigp#6179) Co-authored-by: kevaundray <[email protected]>

jimmygchen mentioned this pull request Aug 1, 2024

Add PeerDAS KZG lib integration (construction & KZG verification) #6212

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version #5941

chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version #5941

kevaundray commented Jun 17, 2024 •

edited

Loading

kevaundray Jun 17, 2024

jimmygchen Jul 7, 2024

kevaundray Jun 17, 2024

kevaundray Jun 17, 2024

kevaundray Jun 17, 2024

kevaundray Jun 17, 2024

kevaundray Jun 17, 2024

kevaundray Jun 17, 2024

jimmygchen commented Jun 19, 2024

kevaundray commented Jun 20, 2024

kevaundray commented Jun 27, 2024

jimmygchen commented Jun 28, 2024

kevaundray commented Jun 28, 2024

jimmygchen Jul 7, 2024

jimmygchen commented Jul 8, 2024

kevaundray commented Jul 8, 2024

jimmygchen commented Jul 8, 2024

jimmygchen commented Jul 9, 2024 •

edited

Loading

jimmygchen commented Jul 9, 2024

jimmygchen commented Jul 9, 2024

	.expect("expected cell to have size {BYTES_PER_CELL}. This should be guaranteed by the `FixedVector type"))
	.expect("expected cell to have size {BYTES_PER_CELL}. This should be guaranteed by the `FixedVector` type"))

chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version #5941

chore: add rust PeerdasKZG crypto library for peerdas functionality and rollback c-kzg dependency to 4844 version #5941

Conversation

kevaundray commented Jun 17, 2024 • edited Loading

Issue Addressed

Proposed Changes

Additional Info

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimmygchen commented Jun 19, 2024

kevaundray commented Jun 20, 2024

kevaundray commented Jun 27, 2024

jimmygchen commented Jun 28, 2024

kevaundray commented Jun 28, 2024

Choose a reason for hiding this comment

jimmygchen commented Jul 8, 2024

kevaundray commented Jul 8, 2024

jimmygchen commented Jul 8, 2024

jimmygchen commented Jul 9, 2024 • edited Loading

jimmygchen commented Jul 9, 2024

jimmygchen commented Jul 9, 2024

kevaundray commented Jun 17, 2024 •

edited

Loading

jimmygchen commented Jul 9, 2024 •

edited

Loading