Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Prover][Test] Broker failure: system must recover if the ProvingBroker goes down #9540

Closed
Tracked by #8495
alexghr opened this issue Oct 29, 2024 · 0 comments · Fixed by #11122
Closed
Tracked by #8495

[Prover][Test] Broker failure: system must recover if the ProvingBroker goes down #9540

alexghr opened this issue Oct 29, 2024 · 0 comments · Fixed by #11122
Assignees
@alexghr
Labels
team-alpha
Milestone
TestNet

Comments

@alexghr
Copy link
Contributor

alexghr commented Oct 29, 2024 •
edited
Loading

Scenario:

  1. start proving a block
  2. accumulate some in proving jobs
  3. shut down ProvingBroker
  4. wait 5 minutes
  5. start ProvingBroker

Expected:

  • orchestrator retries requests to ProvingBroker until it recovers
  • agents retry requests to ProvingBroker until it recovers
  • agents do not lose finished proofs that didn't get posted back to the broker
  • after ProvingBroker re-initializes block proving continue from where it left off
@alexghr alexghr mentioned this issue Oct 29, 2024
Open
@alexghr alexghr changed the title Broker failure: can the system recover if the broker goes down for 5 minutes? [Prover][Test] Can the system recover if the ProvingBroker goes down for 5 minutes? Oct 29, 2024
@alexghr alexghr self-assigned this Oct 29, 2024
@alexghr alexghr added this to A3 Oct 29, 2024
@github-project-automation github-project-automation bot moved this to Todo in A3 Oct 29, 2024
@alexghr alexghr added this to the TestNet milestone Oct 29, 2024
@alexghr alexghr changed the title [Prover][Test] Can the system recover if the ProvingBroker goes down for 5 minutes? [Prover][Test] Broker failure: system must recover if the ProvingBroker goes down Oct 29, 2024
@alexghr alexghr linked a pull request Jan 9, 2025 that will close this issue
test: verify proving is resumed after broker crash #11122
Merged
@github-project-automation github-project-automation bot moved this from Todo to Done in A3 Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexghr alexghr

Labels
team-alpha
Projects
A3
Status: Done
Milestone
TestNet
Development

Successfully merging a pull request may close this issue.

test: verify proving is resumed after broker crash AztecProtocol/aztec-packages
1 participant
@alexghr