Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking: Engine revamp #8742

Closed
88 tasks done
mattsse opened this issue Jun 11, 2024 · 3 comments
Closed
88 tasks done

Tracking: Engine revamp #8742

mattsse opened this issue Jun 11, 2024 · 3 comments
Labels
A-engine Related to the engine implementation C-debt A clean up/refactor of existing code C-tracking-issue An issue that collects information about a broad development initiative M-prevent-stale Prevents old inactive issues/PRs from being closed due to inactivity

Comments

@mattsse
Copy link
Collaborator

mattsse commented Jun 11, 2024 •
edited by jenpaff
Loading

Describe the feature

The current engine <-> tree design has several drawbacks

  • sequential processing

  • Tree Read lock that interferes with rpc (pending block access)

  • very complex request processing that requires multiple tree access roundtrips

  • This must be redesigned entirely

This should allow us to support other interfaces like ABCI,Arbitrum or others via the Node Builder

Tasks
Preview Give feedback
  1. Add high level ChainOrchestrator abstraction #8744
    A-blockchain-tree C-enhancement S-stale
  2. In Memory Engine Tree state #8745
    A-blockchain-tree C-debt
    rkrasiuk
  3. Database Migration Thread #8746
    3 of 3
    A-blockchain-tree C-debt
    Rjected
  4. Blockchain Provider tree state integration #8747
    A-blockchain-tree C-debt
    fgimenez
  5. Node Builder integration of new engine implemention #9559
    C-enhancement
    mattsse
  6. rename PipelineHandler #9008
    A-consensus C-debt
    fgimenez
  7. Merge EngineApiRequestHandler and EngineHandler #9009
    mattsse
  8. Implement blockdownloader for p2p #9011
    A-consensus A-networking C-debt
    fgimenez
  9. Implement pipeline BackfillSync #9012
    A-consensus C-debt
    fgimenez
  10. feat: new engine API handler #8559
    A-consensus C-debt C-perf
  11. Add ethereum/engine implementation #9151
    C-enhancement
    fgimenez
  12. Integrate tree and persistence setup in EthService #9576
    A-consensus C-enhancement
    fgimenez
  13. Support --debug.tip in EthService #9577
    C-enhancement
  14. Convert Ethservice into stream #9582
    C-enhancement
    fgimenez
  15. Add new BlockchainProvider type #9607
    C-enhancement
    fgimenez
  16. Integrate memory overlay in canonical state #9614
    C-enhancement
    fgimenez
  17. Add more docs for tree engine #9615
    C-docs S-stale
  18. Rename BlockSource::Database variant #9617
    D-good-first-issue S-blocked
  19. Remove InMemoryState trait #9638
    C-debt
    fgimenez
  20. Integrate chainorchestrator events into node events #9644
    C-enhancement
  21. Improve engine event variants #9671
    C-enhancement
    mattsse
  22. Issue requests for missing blocks on forkchoice update #9672
    C-enhancement
    mattsse
  23. Trigger pipeline sync from engine/tree handler #9673
    C-enhancement
    mattsse
  24. Improve treeevent handling for downloaded blocks #9685
    C-debt C-enhancement
    mattsse
  25. handle tree errors gracefully #9688
    C-debt
    Rjected
  26. Recheck backfill state after we finished pipeline sync #9689
    mattsse
  27. Update sync state status in the tree #9694
    C-enhancement
  28. Implement make canonical #9696
    C-enhancement
  29. add consistency check on startup #9700
    C-enhancement
    joshieDo
  30. Integrate payloadbuilder handler #9705
    C-enhancement
    mattsse
  31. Integrate blockchainprovider2 in ethnodelauncher #9731
    C-enhancement
    fgimenez
  32. Better testing support for engine api #9749
    5 of 5
  33. Add missing checks for canonical safe and finalized blocks #9750
    C-enhancement
  34. Improve in memory block tracking #9752
    S-needs-design
    fgimenez
  35. Support sending messages to EngineApiImpl from itself #9753
    S-needs-design
    mattsse
  36. Provide CanonicalInMemory state as input to EngineHandler #9760
    fgimenez
  37. replace generic stateprovider with boxed trait #9777
    C-enhancement
    fgimenez
  38. Move MemoryOverlayStateProvider to chain-state crate #9778
    C-debt
    fgimenez
  39. Make EngineHandler terminate if underlying channels close #9793
    C-debt
  40. Improve engine observability #9794
    S-stale
  41. Add metrics for inmemory canonical state #9806
    A-observability C-enhancement
    onbjerg
  42. Notify tracker about received engine messages #9807
    mattsse
  43. Add support for merging execution outputs #9812
  44. Improve backfill state tracking #9815
    C-enhancement D-good-first-issue
  45. Make tree settings configurable #9821
    C-enhancement
    fgimenez
  46. reverse MemoryOverlayStateProvider block order #9825
    fgimenez
  47. Add BlockSource::Database variant #9828
  48. Update pending block if new #9829
    mattsse
  49. Remove StateProvider::pending_with_provider trait function #9849
    C-debt
    mattsse
  50. ensure backfill only starts if persistence task finished #9893
    C-bug
    mattsse
  51. Only persist the canonical chain #9897
    C-bug
    fgimenez
  52. check if head is already canonical #9899
    C-bug
    mattsse
  53. Return fatal error if persistence task dropped #9909
    mattsse
  54. Improve persistence action commits #9923
    C-debt
    joshieDo
  55. Support in memory lookups in BlockchainProvider2 #9924
    fgimenez
  56. Improve message tree task message model #9926
    C-debt
    mattsse
  57. Handle inserting disconnected blocks gracefully #9935
    mattsse
  58. Add metrics for in memory tree and canonical memor #9957
    A-observability
    malik672
  59. gracefully handle persistence errors #9958
    C-debt
    Rjected
  60. dont truncate blocks to persist #9978
    C-perf
    fgimenez
  61. Propagate and handle errors in database and static file services #9535
    A-db A-static-files C-enhancement S-stale
  62. Improve missing block downloading #10007
    C-bug
    mattsse
  63. blocks to persist can be empty #10009
    A-observability
    fgimenez
  64. Live sync transition scenario tests #10015
    C-bug C-test
    fgimenez
  65. Improve connecting downloaded blocks #10017
    C-bug C-test
    fgimenez
  66. Improve download observability #10018
    fgimenez
  67. Improve in memory range lookups #10037
    C-enhancement D-good-first-issue
    PelleKrab
  68. Use BtreeMap for block numbers #10038
    C-enhancement D-good-first-issue
  69. Make payloadbuilder emit an executed block #10096
    mattsse
  70. Introduce custom engine APi commands as request variants #10097
  71. bubble up download event from engineHandler #10098
  72. add devmode launcher implementation #10104
    C-enhancement D-complex
    malik672
  73. Ensure persistence receiver is dropped when value is received #10115
    C-bug
    fgimenez
  74. remove EngineApiTreeHandler trait #10123
    C-debt
    mvares
  75. Introduce EngineApiKind #10124
    C-enhancement S-stale
  76. Make EthService reusable #10145
  77. Convert pending block to a watch channel #10157
    C-enhancement
    jenpaff
  78. Integrate ExecutedBlock in BuiltPayload #10159
  79. Integrate Engine2 for Optimism #10160
    C-enhancement
    fgimenez
  80. BlockNotfound in treestate warning #10179
    C-bug
  81. Add threshold for how many blocks can be persisted #10187
    C-enhancement
    fgimenez
  82. Tracking: In-memory compliance for BlockchainProvider2 #10186
    11 of 11
    A-blockchain-tree A-db C-tracking-issue
  83. track persistence duration #10220
    D-good-first-issue
    malik672
  84. extract persistence event handling into standalone functions #10221
    D-good-first-issue
    mvares
  85. Properly bubble up InsertBlockFatalError #10219
    A-blockchain-tree C-enhancement
    martinezjorge
  86. Properly handle restart edge cases between ExEx and new engine #10489
    A-blockchain-tree A-exex M-prevent-stale S-needs-design
    shekhirin
  87. Provider returns incorrect or missing data when used externally #8751
    A-static-files C-bug M-prevent-stale
    joshieDo
  88. bug(engine): persisted executed block retrieval #11251
    C-bug
    Rjected

Additional context

No response
@mattsse mattsse added C-enhancement New feature or request C-tracking-issue An issue that collects information about a broad development initiative A-engine Related to the engine implementation labels Jun 11, 2024
@shekhirin shekhirin changed the title Engine revamp tracking issue Tracking: Engine revamp Jun 11, 2024
@shekhirin shekhirin pinned this issue Jun 11, 2024
@Rjected Rjected mentioned this issue Jun 12, 2024
Closed
@gakonst gakonst mentioned this issue Jun 13, 2024
Merged
This was referenced Jun 15, 2024
Closed
Closed
Closed
Merged
Merged
@emhane
Copy link
Member

emhane commented Jun 24, 2024

will solve these issues #8303, especially #8746 and #8745

@emhane emhane added C-debt A clean up/refactor of existing code and removed C-enhancement New feature or request labels Jun 24, 2024
@emhane
Copy link
Member

emhane commented Jun 27, 2024

#8745 and #8746 should solve #7322

This was referenced Jun 27, 2024
Closed
Closed
@fgimenez fgimenez mentioned this issue Jul 1, 2024
Merged
@Rjected Rjected mentioned this issue Jul 1, 2024
Closed
This was referenced Jul 13, 2024
Closed
Closed
This was referenced Jul 22, 2024
Merged
Merged
@mattsse mattsse mentioned this issue Jul 29, 2024
Merged
@Rjected Rjected unpinned this issue Aug 16, 2024
@Rjected Rjected pinned this issue Aug 16, 2024
@fgimenez fgimenez mentioned this issue Aug 20, 2024
Closed
@onbjerg onbjerg mentioned this issue Aug 22, 2024
Closed
@emhane emhane unpinned this issue Aug 22, 2024
@emhane emhane pinned this issue Aug 22, 2024
@mattsse mattsse mentioned this issue Aug 26, 2024
Closed
@github-actions GitHub Actions
Copy link
Contributor

This issue is stale because it has been open for 21 days with no activity.

@github-actions github-actions bot added the S-stale This issue/PR is stale and will close with no further activity label Sep 23, 2024
@shekhirin shekhirin added C-tracking-issue An issue that collects information about a broad development initiative M-prevent-stale Prevents old inactive issues/PRs from being closed due to inactivity and removed C-tracking-issue An issue that collects information about a broad development initiative S-stale This issue/PR is stale and will close with no further activity labels Sep 23, 2024
@mattsse mattsse modified the milestone: engineV2 default Oct 3, 2024
@mattsse mattsse closed this as completed Oct 17, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in Reth Tracker Oct 17, 2024
@onbjerg onbjerg unpinned this issue Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-engine Related to the engine implementation C-debt A clean up/refactor of existing code C-tracking-issue An issue that collects information about a broad development initiative M-prevent-stale Prevents old inactive issues/PRs from being closed due to inactivity
Projects
Reth Tracker
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@shekhirin @mattsse @emhane