Skip to content
This repository has been archived by the owner on Nov 6, 2020. It is now read-only.

sync: prevent ancient block import queue becoming full #9754

Closed
ascjones opened this issue Oct 15, 2018 · 2 comments
Closed

sync: prevent ancient block import queue becoming full #9754

ascjones opened this issue Oct 15, 2018 · 2 comments
Labels
M4-core ⛓ Core client code / Rust.
Milestone

Comments

@ascjones
Copy link
Contributor

As detailed in #9531, the ancient block import queue fills up frequently which was causing the ancient blocks sync to retract. That fix should prevent the retractions, but another issue is that the queue still fills up fairly frequently, probably due to contention with the NewBlocks sync or slow writing to disk (need to investigate). The effect of this is that lots of downloaded ancient blocks are discarded (the ones that can't fit in the queue) and they have to be downloaded again. The problem gets worse as we get into the 3_000_000+ range, downloading blocks faster than we can import them.

Brainstorming solutions

One solution would be to reuse the mechanism of the NewBlocks sync which pauses the downloading of blocks if the queue is full. For that to work we would have to figure out a way to keep the downloaded blocks which couldn't go in the queue. Could make the queue unbounded (like the tx and main block import queue), allowing the current batch to be added in its entirety, but with a check against queue_full which would pause the downloading of more blocks.

A potentially simpler alternative might be to change the number of blocks to request per round to be the same as the queue capacity, and only move to the next round once the queue is empty. Downside is that would slow down the syncing process, but that may not be much of a problem. Could mitigate that by checking the queue is half (or some %) full and then requesting that amount of blocks. Might still occasionally get a full queue because of a race condition but we can handle that.

Another avenue might be to investigate whether we can improve the import speed, is it because of contention with NewBlocks sync or just limited by disk IO.

Pinging @tomusdrw as we've chatted about this before, if you have any thoughts.

@ascjones ascjones added the M4-core ⛓ Core client code / Rust. label Oct 15, 2018
@tomusdrw
Copy link
Collaborator

I think this fits a bit into more general way how we could re-work importing stuff from the network. I believe it would be worth to try mpsc::sync_channel for all the cases we have - the idea is that we would have one thread that handles the import and networking tasks would just push stuff to the channel - in case the bound is reached (to be determined experimentally), the channel starts to block the threads and wait for completion (so pretty much back-pressures the sync process).
This has to be carefuly tested though, cause we don't really want all IoWorker threads to be blocked with some tasks trying to import stuff to the DB (essentially we would return to issues we had before this process was done asynchronous). Potentially this could be reworked further to specialise the IoWorker threads and have different behaviour for "write" and "read-only" tasks - i.e. we really want the networking to be able to respond to all requests and keep connections running (best effort) while we are busy importing blocks, but we want to back-pressure importing new blocks/transactions if we are struggling with the current ones we have. That would require some kind of prioritization mechanism for different types of tasks that are run on the IoWorkers (i.e. opening/closing connections, responding to ping, handling different kinds of packets would need to have different priorities)

@5chdn 5chdn added this to the 2.3 milestone Oct 29, 2018
@5chdn 5chdn modified the milestones: 2.3, 2.4 Jan 10, 2019
@5chdn 5chdn modified the milestones: 2.4, 2.5 Feb 21, 2019
@soc1c soc1c modified the milestones: 2.5, 2.6 Apr 2, 2019
@ordian ordian modified the milestones: 2.6, 2.7 Jul 12, 2019
@adria0
Copy link

adria0 commented Jul 27, 2020

Closing issue due to its stale state.

@adria0 adria0 closed this as completed Jul 27, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
M4-core ⛓ Core client code / Rust.
Projects
None yet
Development

No branches or pull requests

6 participants