Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: free the blobs #5181

Merged
merged 4 commits into from
Aug 9, 2023
Merged

feat: free the blobs #5181

merged 4 commits into from
Aug 9, 2023

Conversation

g11tech
Copy link
Contributor

@g11tech g11tech commented Feb 20, 2023 •
edited
Loading

Free the blobs

the branch is now functional! (cleanup and some blob gossip validations still wip)

Below are the logs of a local devnet (with a corresponding ethereumjs branch) where peer1 is producing blobs
And peer2 syncs block and blobs

  1. Initially via req/resp to get Synced
  2. Then via gossip to stay Synced
    image
  3. Unknown Parent root sync using beacon/blob side cars by root
    image

ref:

TODOS:

  • modified block/blob production
  • modified blob dbs
  • handled block/blob gossip
  • marry blob and block gossip for BlockInput
  • updated references
  • fixed gossip types
  • updated fetch blocksMaybeBlobsbyRange
  • updated fetch blocksMaybeBlobsbyRoot
  • handle block/blob archival to finalizeddbs
  • blob publish
  • fix block publish
  • handle onBlobSideCarByRange
  • handle onBlockSideCarbyRoot
  • prune gossip blockinput cache
  • handle genesis state blobs db handling
  • validator blob signing and publishing
    • modify get produced blobs
    • sign and propagate
  • integrate c-kzg with computeBlobKzgProof
    • update interfaces

TODO:

This was referenced Mar 17, 2023
Merged
Closed
@g11tech g11tech force-pushed the g11tech/move-data-availabilty branch from 387c252 to 0778d1f Compare March 18, 2023 17:00
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch 2 times, most recently from aab1541 to 866f177 Compare March 18, 2023 18:36
@g11tech g11tech force-pushed the g11tech/move-data-availabilty branch from 0778d1f to 79a94b7 Compare March 19, 2023 13:09
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 53804e6 to 60f5c79 Compare March 19, 2023 13:12
@g11tech g11tech force-pushed the g11tech/move-data-availabilty branch from 79a94b7 to 657e1aa Compare March 24, 2023 18:19
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 3b52d7b to 3d99502 Compare March 24, 2023 18:23
@g11tech g11tech force-pushed the g11tech/move-data-availabilty branch from 657e1aa to 4047185 Compare April 1, 2023 13:45
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 4886dfd to 84043b2 Compare April 1, 2023 14:15
@g11tech g11tech force-pushed the g11tech/move-data-availabilty branch from 4047185 to f9049ca Compare April 9, 2023 11:54
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from f2e86c2 to 2c206e0 Compare April 9, 2023 12:34
@g11tech g11tech force-pushed the g11tech/move-data-availabilty branch from f9049ca to 06bb540 Compare April 22, 2023 09:28
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 4d36e75 to 94a2fe7 Compare May 22, 2023 08:35
@g11tech g11tech mentioned this pull request May 23, 2023
Merged
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 6d61c4c to 1cafb88 Compare May 26, 2023 15:39
@g11tech g11tech mentioned this pull request May 29, 2023
Merged
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch 2 times, most recently from 1969f70 to 496a426 Compare June 13, 2023 07:50
This was referenced Jun 17, 2023
Merged
Merged
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch 2 times, most recently from 27fdd86 to 6c23a8b Compare June 23, 2023 09:02
@g11tech g11tech mentioned this pull request Jun 23, 2023
Merged
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from a844288 to 26bbbe4 Compare July 27, 2023 12:22
@g11tech g11tech marked this pull request as ready for review July 28, 2023 06:49
@g11tech g11tech requested a review from a team as a code owner July 28, 2023 06:49
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 6e82771 to 7b0e9f2 Compare August 1, 2023 11:29
twoeths
twoeths reviewed Aug 2, 2023
View reviewed changes
Copy link
Contributor

@twoeths twoeths left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great, I dropped some minor comments

@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 7b0e9f2 to 966d55b Compare August 2, 2023 14:25
g11tech added 4 commits August 6, 2023 18:06
@g11tech
Free the blobs
c2c4648 
fix the types

rejig the new constants in params

add comment for cleanup

update reqresp

fix api package

rename blobs repo

commit the wip modifications

further appropriate renaming

further references update

further reference updates

continue refac

fix reqresp build

further refac

further refac

fix api

fix db interface

fix beacondb alloc

build

fix api

improve blob verificaion

correct validation call

fixes

fix the produce block/blobs flow

reduce diff

blob gossip validation

update validations

cleanup block vali

reduce diff

handle gossip of block and blob

fix test for timebeing

modify publishing flow

fix import flow

onsidecarbyrange fix and some type fixes

fix sidecars by root

prune blockinput cache

fix kzg interface

small renaming

interface rename

fix fetch blockmaybeblobs by range test

fix build lint issues for now

c-kzg version fix

FullOrBlindedBlobSidecar changes

fix tests

complete the blob publishing flow

fix test

get the single node run functional

get the gossip blob flow working

fix peer syncing using req/resp

fix sidecar by root check

refactor blobsidecars hotdb and remove archive

add blob gossip validation flow

fix topic

fix the validation condition

add blob validation and test various sync modes

fix tests

rebase fixes

enable deneb spec tests

make blobsbyroot multi block

fixes

cleanup defunt builder endpoint

archive blobs post finalization uptill the blob window

serve finalized blobs within the blob prune window

fix test

fix test

lookup in archive as well

cleanup and improvements

rebase fx

Add 4844 sim test and override the field elements per blob

update image

add blob test

add test run in package

start unknown sync and range sync

finalize the sims

change the signing flow

fix test types

fix tests

fix test

lint

update tx type and corresponding ethereumjs image

update c-kzg and use blobs bundle proof

fix test

fix test

merge getblobsbundle into getpayloadv3

update images

fix genesis config

rebase fixes

fix test

update images

fix tests

lint

fix the sidecar request count limit

fix test

lint

rebase fixes

cleanup

fix unit tests

update kzg to big endian

devnet 6 integration

fix passing setup arg

update path

fix tests

fix blobs sidecar by range response

cleanup blob import vals

reduce diff

cleanups

disable blob sim tests till spec stablizes

handle blob bytes in blockinput

fixes
@g11tech
apply feedback
12eeaf8
@g11tech
add todo for pre-emptive block/blobs pull
f9f3fd9
@g11tech
rebase fix
2c70d31
@g11tech g11tech force-pushed the g11tech/free-the-blobs branch from 3fd6b2f to 2c70d31 Compare August 6, 2023 12:38
dapplion
dapplion approved these changes Aug 9, 2023
View reviewed changes
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@g11tech g11tech merged commit 6d183c9 into unstable Aug 9, 2023
@g11tech g11tech deleted the g11tech/free-the-blobs branch August 9, 2023 12:52
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.11.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kevaundray kevaundray kevaundray left review comments

@wemeetagain wemeetagain wemeetagain left review comments

@dapplion dapplion dapplion approved these changes

@twoeths twoeths twoeths approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@g11tech @wemeetagain @twoeths @dapplion @kevaundray