Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import disks from ESXi to XCP-ng using XVA format #7323

Merged
merged 4 commits into from
Jan 30, 2024
Merged

Conversation

fbeauchamp
Copy link
Collaborator

@fbeauchamp fbeauchamp commented Jan 18, 2024

DO NOT SQUASH, review by commits.

Description

Changes :

  1. load base disk then load snapshots, instead of loading a synthetic vmdk built from the base disk + snapshot
  2. use xva import endpoint instead of vhd endpoint to import base disks into XCP:
  • read base disk once instead of twice, de facto building thin disk for free
  • use less memory
  • smaller blocks means less empty blocks transfered
  1. batch read when handling a sesparse snapshot ( esxi 6.5+)
  2. don't read parent block if the current block of the snapshot is already full

Advice :

  • please intall a xo in your esxi for the migration : most small read are done from XO to the esxi ,and theses are very sensible to latency

results

  • XO installed as a VM directly on the ESXI8
  • XCP-ng in local network over a 1Gbps link ( so 116MB/s max)
  • speed limit of reading a flat.vmdk without doing anything ~330MB/s
  • 1 VM , stopped, with 1 disk, 1 snapshot
  • import duration on this branch : 4m46
  • import duration on master : 71m25

network speed on the XO during the base disk transfer (read from esxi, write to xcp-ng)
image (6)

network speed on the XO during the import of a snapshot of 1 VM, with 1 disk, with 1 snapshot, (read from esxi, write to xcp-ng)
image (7)

review by commits

Checklist

  • Commit
    • Title follows commit conventions
    • Reference the relevant issue (Fixes #007, See xoa-support#42, See https://...)
    • If bug fix, add Introduced by
  • Changelog
    • If visible by XOA users, add changelog entry
    • Update "Packages to release" in CHANGELOG.unreleased.md
  • PR
    • If UI changes, add screenshots
    • If not finished or not tested, open as Draft

@fbeauchamp fbeauchamp force-pushed the xva_generation branch 4 times, most recently from 3b63b4b to 6787d0b Compare January 19, 2024 14:22
@fbeauchamp fbeauchamp marked this pull request as ready for review January 19, 2024 14:32
@fbeauchamp fbeauchamp force-pushed the xva_generation branch 2 times, most recently from 61f51c9 to b678183 Compare January 19, 2024 14:36
@fbeauchamp fbeauchamp requested a review from julien-f January 22, 2024 09:07
@julien-f julien-f changed the title feat(import): xva import Import disks from ESXi to XCP-ng using XVA format Jan 25, 2024
@fbeauchamp fbeauchamp force-pushed the xva_generation branch 2 times, most recently from ff3c647 to 168fb6e Compare January 29, 2024 08:17
@fbeauchamp fbeauchamp force-pushed the xva_generation branch 2 times, most recently from 01d16a9 to 59ee93a Compare January 29, 2024 13:12
@fbeauchamp fbeauchamp requested a review from julien-f January 29, 2024 13:15
@fbeauchamp fbeauchamp requested a review from julien-f January 30, 2024 08:41
@julien-f julien-f removed their request for review January 30, 2024 09:48
@fbeauchamp fbeauchamp force-pushed the xva_generation branch 2 times, most recently from ce1bf25 to 5386afc Compare January 30, 2024 10:30
@fbeauchamp
Copy link
Collaborator Author

@julien-f I finish testing it, and it gave me the same raw as master
I will do one more rebase and it is ready to merge

@julien-f julien-f merged commit 3ab2a83 into master Jan 30, 2024
1 check passed
@julien-f julien-f deleted the xva_generation branch January 30, 2024 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants