Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node crashes on quick restart after being killed #2838

Closed
ulope opened this issue Oct 18, 2018 · 4 comments · Fixed by #2884
Closed

Node crashes on quick restart after being killed #2838

ulope opened this issue Oct 18, 2018 · 4 comments · Fixed by #2884
Assignees
@hackaugusto
Labels
Severity / Minor

Comments

@ulope
Copy link
Collaborator

ulope commented Oct 18, 2018

Problem Definition

(This may be parity related, haven't tested with Geth yet.)

Steps to reproduce:

  • Kill a node (with SIGKILL, -9) right after a new block was received
  • Restart node within a few seconds

The node crashes with a ValueError from within web3:

ValueError: {'code': -32000, 'message': 'One of the blocks specified in filter (fromBlock, toBlock or blockHash) cannot be found', 'data': '0x8b2045'}

Full logs
As you can see the data attribute in the error is increasing and seems to be a block number.

  • 0x8b2045 -> 9.117.765
  • 0x8b2047 -> 9.117.767
  • 0x8b2048 -> 9.117.768

This error persists for a while but disappears after a few minutes.
@hackaugusto
Copy link
Contributor

Maybe this was fixed by #2671

@ulope
Copy link
Collaborator Author

ulope commented Oct 19, 2018 •
edited
Loading

@hackaugusto Unfortunately not, this was with latest master as of yesterday evening (e0f3e6f) which already includes that PR.

@ulope ulope mentioned this issue Oct 24, 2018
Closed
@hackaugusto hackaugusto self-assigned this Oct 24, 2018
@hackaugusto
Copy link
Contributor

Ahhh, I think I know what is the problem:

raiden/raiden/utils/filters.py

Line 111 in 97820b6

self._last_block + 1,

hackaugusto added a commit to hackaugusto/raiden that referenced this issue Oct 24, 2018
@hackaugusto
Fix race condition with new blocks and filters
6eb02a9 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
@hackaugusto hackaugusto mentioned this issue Oct 24, 2018
Merged
palango pushed a commit to hackaugusto/raiden that referenced this issue Oct 28, 2018
@hackaugusto @palango
Fix race condition with new blocks and filters
df2d076 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
@LefterisJP
Copy link
Contributor

As discussed in Rocketchat this should no longer be happening after #2895 which now updates the block state only with confirmed blocks.

LefterisJP pushed a commit to LefterisJP/raiden that referenced this issue Nov 7, 2018
@hackaugusto @LefterisJP
Fix race condition with new blocks and filters
7f5720e 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
LefterisJP pushed a commit that referenced this issue Nov 8, 2018
@hackaugusto @LefterisJP
Fix race condition with new blocks and filters
3b65367 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes #2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
395f625 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
175114b 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
7155b8c 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
f5979b8 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
bc5eb39 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
4b68311 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
772b559 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
1c7893d 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
4383cf3 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
d2cdb62 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
bdcf1ff 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 18, 2019
@hackaugusto
Fix race condition with new blocks and filters
fe460f0 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 25, 2019
@hackaugusto
Fix race condition with new blocks and filters
129ae35 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 25, 2019
@hackaugusto
Fix race condition with new blocks and filters
563ddd8 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 25, 2019
@hackaugusto
Fix race condition with new blocks and filters
a24a41b 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 25, 2019
@hackaugusto
Fix race condition with new blocks and filters
a425abb 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 25, 2019
@hackaugusto
Fix race condition with new blocks and filters
74ffaad 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
hackaugusto added a commit to hackaugusto/raiden that referenced this issue Jan 25, 2019
@hackaugusto
Fix race condition with new blocks and filters
003473a 
The race happened under this circunstance:

- A node learns about a new block, updates its state, then crashes
- On restart, the block number is recovered, the filters are
  installed with the latest known block.
- The race: The node finishes the above before a new block is mined
- The bug: The filter is polled during start of the RaidenService, by
  calling the AlarmTask.first_run, which always executes the
  callbacks, eventually using the StateFilter's to poll for new events
  from a block in the future.

The fix was to give the latest known block number to the alarm task in
the as an argument for first_run, and only execute the callbacks if
there is a new block.

fixes raiden-network#2838
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hackaugusto hackaugusto

Labels
Severity / Minor
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix race condition with new blocks and filters hackaugusto/raiden
3 participants
@ulope @hackaugusto @LefterisJP