Executable EIP-4844 specs #2901

hwwhww · 2022-05-24T16:21:30Z

In this PR:

Added kzg.py for generating testing trusted setup.
- Ref: @vbuterin's https://github.com/ethereum/research/blob/8f084630528ba33d92b2bc05edf5338dd193c6f1/trusted_setup/trusted_setup.py
Renamed/reused config
- SHARDING_FORK_VERSION -> EIP4844_FORK_VERSION
- SHARDING_FORK_EPOCH -> EIP4844_FORK_EPOCH
Added test-eip4844 CI job
Fixed tests
- Right now, both EIP4844 and Capella's pre-fork is Bellatrix. This PR updated the testing tool to make it compatible.

TODOs:

(Not in this PR) Handle mainnet config FIELD_ELEMENTS_PER_BLOB: 4096
- Since (i) mainnet trusted setup is not complete (ii), it would really slow to generate trusted setup with large length with py-ecc and (iii) spec builder dependency, right now mainnet KZG_SETUP_G2 and KZG_SETUP_LAGRANGE are actually minimal config values.

hwwhww · 2022-05-31T16:43:17Z

@asn-d6 @protolambda
Do you mind reviewing the specs and kzg.py parts?

asn-d6

This looks great, after an initial review!

I'm gonna be working on integrating ethereum/EIPs#5088 into the consensus-specs this week, so there will be a bunch of new things after that's done. When it's time, I will leave a new comment here with some more information.

Cheers!

asn-d6 · 2022-06-13T11:59:17Z

setup.py

+    @classmethod
+    def sundry_functions(cls) -> str:
+        return super().sundry_functions() + '''
+# TODO: for mainnet, load pre-generated trusted setup file to reduce building time.


Here is a py_ecc-based insecure trusted setup you might want to use for this purpose: https://github.com/asn-d6/kzgverify/blob/playground/trusted_setup.py (and corresponding json files on same directory)

tests/core/pyspec/eth2spec/test/eip4844/unittests/test_kzg.py

tests/core/pyspec/eth2spec/utils/kzg.py

asn-d6 · 2022-06-14T15:43:13Z

Hello @hwwhww! I have created the PR for the consensus-side verification logic that uses KZG proofs. Please see #2915

I imagine that this would warrant a bunch of changes to the code of this PR. In particular:

Implement the math/polynomial functions introduced by the PR.
Introduce the ROOTS_OF_UNITY constant list of BLS field elements. You can generate them with code that looks like this https://github.com/ethereum/consensus-specs/blob/danksharding/specs/sharding/polynomial-commitments.md#roots_of_unity
Finally, to make an end-to-end test that goes from block creation all the way to verification, we would need to create a valid kzg_aggregated_proof that can be verified. To do that, we need to create a bunch of blobs and commitments and then basically faithfully reproduce the steps performed in verify_blobs_sidecar() to compute:
- r_powers from the various blobs and commitments
- aggregated_poly from the individual blobs
- aggregated_poly_commitment from the corresponding commitments
- x from the aggregated_poly and aggregated_poly_commitment
finally, we use those objects to create a valid KZG proof for that aggregated poly at x That proof is the one that the tests will verify.

Please let me know if the above did not make any sense!

hwwhww · 2022-06-29T17:15:51Z

Hey @asn-d6

Updated some quick fixes after the #2915 patch in 60fdcac.

Moved vector_lincomb to polynomial-commitments.md
Fixed function signatures: replace List with Sequence
To use hash_to_bls_field properly, created BlobsAndCommmitments and PolynomialAndCommitment SSZ containers.
Add custom type Polynomial
To exclude the snippet sample code in p2p spec & v-guide, remove “python” from the code block for now.

…x_lincomb`

hwwhww · 2022-07-01T15:42:38Z

Found more bugs: 33829dd

Fixed offset calculation in tx_peek_blob_versioned_hashes
Fixed lincomb return value type: in Pyspec, r is an optimized_bls12_381_FQ object so we need serialization. G1_to_bytes48 is the function that we used in BLS pubkey serialization.
Fixed BlobsSidecar sample code: / -> Python //

hwwhww · 2022-07-01T18:31:26Z

@asn-d6

In validator guide, it described how to pack the blobs into a BlobsSidecar object. However, BlobsSidecar.kzg_aggregated_proof value is missing here.

Should we add notes on how to generate proof to the validator guide?

asn-d6 · 2022-07-02T09:06:59Z

@asn-d6

In validator guide, it described how to pack the blobs into a BlobsSidecar object. However, BlobsSidecar.kzg_aggregated_proof value is missing here.

Should we add notes on how to generate proof to the validator guide?

Yep! I think that's needed!

The notes in the last point of #2901 (comment) could serve as a decent guide here I think.

asn-d6 · 2022-07-02T09:30:36Z

Found more bugs: 33829dd

Good catches!

2. Fixed `lincomb` return value type: in Pyspec, `r` is an `optimized_bls12_381_FQ` object so we need serialization. `G1_to_bytes48` is the function that we used in BLS pubkey serialization.

I'm a bit sad that we have to do this ugly cast KZGCommitment(kzg.G1_to_bytes48(r)) in the spec. I understand why it's there and I don't have a solution atm, but I'm just mentioning this because I don't remember similar casts in the BLS parts of the spec.

…ments.md

hwwhww · 2022-07-05T09:22:51Z

Updates:

Add compute_proof_from_blobs and its sub helpers. (Thank @asn-d6 for helping debug!)
Add tests to verify the validator guide.
Add @protolambda explainer on blob_versioned_hashes offset calculation
Renames
1. quotient_kzg -> kzg_proof
2. Parameters. Use fewer abbreviations.
Fix is_data_available
Add int() casting during polynomial computation to avoid SSZ object overflow/underflow.
Add G1_to_bytes48/bytes48_to_G1/G2_to_bytes96/bytes96_to_G2 in some KZG functions.
- Re: Executable EIP-4844 specs #2901 (comment)
  - The BLS API wrappers (IETF BLS signature draft standard v4 implementation) handled it internally so I didn't have to do the conversion in specs. But now we are dealing with low-level computation in specs so things are exposed. These conversions only happen in polynomial-commitments.md currently. Hope that it will be standardized soon and then CL implementors don't have to implement the conversions.

hwwhww · 2022-07-06T09:51:00Z

Update: ec7f607

As discussed, rename most KZGCommitment object kzg -> commitment

blob_kzgs -> blob_commitments
- note: there is also "KZGProof"
blob_to_kzg -> blob_to_commitment
BLOB_COMMITMENT_VERSION_KZG -> BLOB_COMMITMENT_VERSION
kzg_to_versioned_hash -> commitment_to_versioned_hash
[New] verify_kzgs_against_transactions -> verify_commitments_against_transactions

Notes:

I use "commitment" in most function signatures where it is obvious a commitment of blob.
Use full name blob_commitments in BeaconBlockBody field.

hwwhww · 2022-07-06T10:50:15Z

So after discussing with @vbuterin @asn-d6 @GottfriedHerold, we realized that "blob commitment" could be confused with versioned hash.

247850a update:

use full name "blob_kzg_commitments".
BLOB_COMMITMENT_VERSION_KZG -> VERSIONED_HASH_VERSION_KZG.

asn-d6 · 2022-07-06T10:01:00Z

specs/eip4844/polynomial-commitments.md

+#### `compute_kzg_proof`
+
+```python
+def compute_kzg_proof(polynomial: Sequence[BLSFieldElement], x: BLSFieldElement) -> KZGProof:


I think it might be a good idea to document here that the polynomial is in evaluation form

asn-d6 · 2022-07-06T10:14:05Z

specs/eip4844/polynomial-commitments.md

+    polynomial = [int(i) for i in polynomial]
+
+    # Convert `polynomial` to coefficient form
+    assert pow(ROOTS_OF_UNITY[1], len(polynomial), BLS_MODULUS) == 1


Now that FFT has penetrated into the core of the spec, it might be worth modding our FFT function to do the inverse FFT inside of it, instead of doing it at the caller.

For example see the wrapper fft() function here https://github.com/ethereum/research/blob/master/mimc_stark/fft.py#L31 that internally calls _fft() which is what we have here (we don't need the simple_ft() thing).

Also, since FFT is not exactly polynomial specific it might be worth introducing an FFT section in this file just for FFT stuff.

After discussion, we don't actually need FFT to make KZG proofs from polynomials in evaluation form.
Please see this branch: https://github.com/asn-d6/consensus-specs/tree/eip4844-exe (pending review)

asn-d6 · 2022-07-06T10:42:43Z

tests/core/pyspec/eth2spec/utils/kzg.py

+    return pow(PRIMITIVE_ROOT_OF_UNITY, (BLS_MODULUS - 1) // length, BLS_MODULUS)
+
+
+def compute_roots_of_unity(field_elements_per_blob):


Might be better to replace field_elements_per_blob with order here and make this function less-application-specific.

We were using `x` as the evaluation point variable in polynomial functions. That can be confused with the indeterminate variable `x` of the polynomial. Hence the convention now is: - `x` for the indeterminate variable of a polynomial - `z` for the point we evaluate it at - `y` for the evaluation result

Previously we were using IFFT to go from evaluation form to coefficient form before computing the KZG proof in the traditional way. We can instead do the KZG proof division directly in evaluation form by using point-by-point division: https://dankradfeist.de/ethereum/2021/06/18/pcs-multiproofs.html#dividing-when-one-of-the-points-is-zero Two fine points: - We need to check that the denominator of the division is not zero. It's very unlikely because `z` comes out of Fiat-Shamir, but we still add an assert to make sure. - We need to shift our main polynomial by `p(z)` as in the traditional KZG proof formula. We can avoid this in coefficient form because with long division we can extract the quotient polynomial and ignore the remainder. However that's not possible when doing point-by-point division in evaluation form and hence we need to ensure that there is no remainder by shifting by `p(z)`.

asn-d6 · 2022-07-13T10:28:05Z

Made an attempt at squashing and rebasing this PR: #2937
Pending review.

hwwhww · 2022-07-13T12:31:59Z

closing in favor of #2937

Executable EIP-4844 specs (PR #2901 squashed and rebased)

Trying to include EIP-4844 specs to pyspec

9811b94

hwwhww added the Deneb was called: eip-4844 label May 24, 2022

hwwhww added 4 commits May 31, 2022 22:15

WIP. Add kzg.py

af7898e

Rename SHARDING_FORK_VERSION to SHARDING_FORK_VERSION and fix tests

238464d

Add test-eip4844 CI job

29b27b3

Fix lint

15fe4c9

hwwhww force-pushed the eip4844-exe branch from ef02921 to 15fe4c9 Compare May 31, 2022 16:22

asn-d6 reviewed Jun 13, 2022

View reviewed changes

hwwhww added 2 commits June 20, 2022 21:05

Merge branch 'dev' into eip4844-exe

edb3894

PR feedback from @asn-d6

7324b90

asn-d6 mentioned this pull request Jun 23, 2022

Optimizing EIP-4844 block validation (using KZG proofs) #2915

Merged

hwwhww added 2 commits June 29, 2022 20:26

Merge branch 'dev' into eip4844-exe

1a3929d

Fix bugs and include all EIP-4844 files to executable scope

60fdcac

hwwhww force-pushed the eip4844-exe branch from 01b807a to 60fdcac Compare June 29, 2022 17:12

hwwhww added 2 commits June 30, 2022 21:30

To avoid confusion with SSZ vector, rename vector_lincomb -> `matri…

b8cbfe5

…x_lincomb`

Fix eip4844 state transition functions

33829dd

hwwhww force-pushed the eip4844-exe branch from 499226c to 33829dd Compare July 1, 2022 15:34

Add validator unittest to check verify_blobs_sidecar.

79b4d77

hwwhww force-pushed the eip4844-exe branch from 7a820ef to 79b4d77 Compare July 4, 2022 16:56

hwwhww added 4 commits July 5, 2022 01:17

Fix matrix_lincomb: add int() casting to avoid SSZ overflow/underflow

13d7c06

Fix is_data_available return value

41c5646

Add KZGProof generation helpers to validator.md and polynomial-commit…

23afa60

…ments.md

Function arguments clean up. Add a link of proto's explainer.

f16ca66

hwwhww marked this pull request as ready for review July 5, 2022 09:22

hwwhww added 3 commits July 5, 2022 17:25

Fix typo compute_kzg_single -> compute_kzg_proof

0f17a95

Remove redundant BLS_MODULUS

71bb4bf

Fix test

36cdab5

hwwhww changed the title ~~Trying to include EIP-4844 specs to pyspec~~ Executable EIP-4844 specs Jul 5, 2022

Renaming KZGCommitment object kzg -> commitment

ec7f607

Use full name kzg_commitment instead of commitment

247850a

asn-d6 reviewed Jul 6, 2022

View reviewed changes

asn-d6 and others added 3 commits July 13, 2022 12:26

Fix ToC

d4af93d

asn-d6 mentioned this pull request Jul 13, 2022

Executable EIP-4844 specs (PR #2901 squashed and rebased) #2937

Merged

hwwhww closed this Jul 13, 2022

asn-d6 added a commit that referenced this pull request Jul 16, 2022

Merge pull request #2937 from asn-d6/eip4844-exe-squashed-and-rebased

9e4f386

Executable EIP-4844 specs (PR #2901 squashed and rebased)

asn-d6 mentioned this pull request Sep 12, 2022

Synchronize EIP-4844 cryptography with consensus specs ethereum/EIPs#5649

Merged

hwwhww deleted the eip4844-exe branch December 1, 2022 16:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Executable EIP-4844 specs #2901

Executable EIP-4844 specs #2901

hwwhww commented May 24, 2022 •

edited

Loading

hwwhww commented May 31, 2022

asn-d6 left a comment

asn-d6 Jun 13, 2022

asn-d6 commented Jun 14, 2022

hwwhww commented Jun 29, 2022

hwwhww commented Jul 1, 2022

hwwhww commented Jul 1, 2022

asn-d6 commented Jul 2, 2022

asn-d6 commented Jul 2, 2022

hwwhww commented Jul 5, 2022 •

edited

Loading

hwwhww commented Jul 6, 2022

hwwhww commented Jul 6, 2022

asn-d6 Jul 6, 2022

asn-d6 Jul 6, 2022 •

edited

Loading

asn-d6 Jul 12, 2022

asn-d6 Jul 6, 2022

asn-d6 commented Jul 13, 2022

hwwhww commented Jul 13, 2022

		return pow(PRIMITIVE_ROOT_OF_UNITY, (BLS_MODULUS - 1) // length, BLS_MODULUS)


		def compute_roots_of_unity(field_elements_per_blob):

Executable EIP-4844 specs #2901

Executable EIP-4844 specs #2901

Conversation

hwwhww commented May 24, 2022 • edited Loading

hwwhww commented May 31, 2022

asn-d6 left a comment

Choose a reason for hiding this comment

asn-d6 Jun 13, 2022

Choose a reason for hiding this comment

asn-d6 commented Jun 14, 2022

hwwhww commented Jun 29, 2022

hwwhww commented Jul 1, 2022

hwwhww commented Jul 1, 2022

asn-d6 commented Jul 2, 2022

asn-d6 commented Jul 2, 2022

hwwhww commented Jul 5, 2022 • edited Loading

hwwhww commented Jul 6, 2022

hwwhww commented Jul 6, 2022

asn-d6 Jul 6, 2022

Choose a reason for hiding this comment

asn-d6 Jul 6, 2022 • edited Loading

Choose a reason for hiding this comment

asn-d6 Jul 12, 2022

Choose a reason for hiding this comment

asn-d6 Jul 6, 2022

Choose a reason for hiding this comment

asn-d6 commented Jul 13, 2022

hwwhww commented Jul 13, 2022

hwwhww commented May 24, 2022 •

edited

Loading

hwwhww commented Jul 5, 2022 •

edited

Loading

asn-d6 Jul 6, 2022 •

edited

Loading