feat(backups): write and merge block concurrency are now configurable #6416
Conversation
fbeauchamp
force-pushed
the
feat_add_setting_merge_concurrency
branch
from
95d39df to
996e538
Compare
| # settings when using Vhd directories ( s3 , encryption ) | ||
| # you should use 'none' if your fs is already compressed | ||
| # changing this setting will generate new full backups | ||
| vhdDirectoryCompression = 'brotli' | ||
|
|
||
| # how many block can be merged in parallel per backup running | ||
| # increase to increase performance, reduce if you have timeout during merge | ||
| mergeBlockConcurrency = 2 | ||
|
|
||
| # how many block can be uploaded in parallel | ||
| # increase to in rease performance, reduce if you have timeout or memory error during transfer | ||
| writeBlockConcurrency = 16 | ||
|
|
There was a problem hiding this comment.
As these settings are for a libraries and are used in two different apps (xo-server & xo-proxy), I feel like we should provide defaults values and document them somewhere else.
I think I will merge your PR as it is and then move them directly in a centralized, would that be ok for you?
There was a problem hiding this comment.
Is "2" enough as default? I think we tried with 4 with success and it was "fast enough" vs 2 (when we tested on Cécile's S3). Or maybe it was 2, I don't remember exactly 🤔
There was a problem hiding this comment.
@julien-f there is default values in the code and documentation in the config file seems like a good start. Where do you want to put the default and the documentation ?
@olivierlambert we tested 4 , but the results came after the patch release.
There was a problem hiding this comment.
Should we change that to 4 then?
There was a problem hiding this comment.
I think we should let this as is in this PR that only exposez existing values
It will be easier to have some users test it by changing the configuration and optimize the default values later.
There was a problem hiding this comment.
@fbeauchamp the config file is not good, because it's in xo-server and not in the lib that really uses these entries. Also, this does not make the settings easily discoverable for xo-proxy.
I think a dedicated @xen-orchestra/backups/config.js file would be a good first answer.
Co-authored-by: Julien Fontanet <[email protected]>
julien-f
changed the title
Check list
Fixes #007orSee xoa-support#42)CHANGELOG.unreleased.md:${name} v${new version})cron/parse.spec.js)xo-serverAPI changes, the corresponding test has been added to/updated onxo-server-testProcess
WiP:(Work in Progress) if not ready to be mergedFrom the Four Agreements: