converter: speed up conversion #352
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imeoer
force-pushed
the
converter-speedup
branch
2 times, most recently
from
d9ec48f to
80ffde8
Compare
Codecov ReportBase: 29.28% // Head: 29.36% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #352 +/- ##
==========================================
+ Coverage 29.28% 29.36% +0.07%
==========================================
Files 38 38
Lines 3811 3811
==========================================
+ Hits 1116 1119 +3
+ Misses 2563 2560 -3
Partials 132 132
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
imeoer
force-pushed
the
converter-speedup
branch
from
80ffde8 to
1bdade8
Compare
imeoer
force-pushed
the
converter-speedup
branch
3 times, most recently
from
16c151e to
be925b2
Compare
imeoer
added a commit
to imeoer/acceleration-service
that referenced
this pull request
Feb 15, 2023
See containerd/nydus-snapshotter#352 It's a internal optimization, so just only bump nydus-snapshotter package to latest version. Signed-off-by: Yan Song <[email protected]>
imeoer
added a commit
to imeoer/image-service
that referenced
this pull request
Feb 15, 2023
Default to enable --type tar-rafs option for nydus-image, which converting OCI tar blob stream into nydus blob directly, eliminating the need to decompress it to a local directory first, thus greatly accelerating the pack process. For compatibility, we still support env `NYDUS_DISABLE_TAR2RAFS=true` variable to disable the optimization. It's a internal feature, so we just only bump acceleration-service and nydus-snapshotter packages to latest version. Related PR: containerd/nydus-snapshotter#352 Signed-off-by: Yan Song <[email protected]>
imeoer
added a commit
to imeoer/image-service
that referenced
this pull request
Feb 15, 2023
Default to enable `--type tar-rafs` option for nydus-image, which converting OCI tar blob stream into nydus blob directly, eliminating the need to decompress it to a local directory first, thus greatly accelerating the pack process. For compatibility, we still support env `NYDUS_DISABLE_TAR2RAFS=true` variable to disable the optimization. It's an internal feature, so we just only bump acceleration-service and nydus-snapshotter packages to the latest version. Related PR: containerd/nydus-snapshotter#352 Signed-off-by: Yan Song <[email protected]>
imeoer
added a commit
to imeoer/image-service
that referenced
this pull request
Feb 15, 2023
Default to enable `--type tar-rafs` option for nydus-image, which converting OCI tar blob stream into nydus blob directly, eliminating the need to decompress it to a local directory first, thus greatly accelerating the pack process. For compatibility, we still support env `NYDUS_DISABLE_TAR2RAFS=true` variable to disable the optimization. It's an internal feature, so we just only bump acceleration-service and nydus-snapshotter packages to the latest version. Related PR: containerd/nydus-snapshotter#352 Signed-off-by: Yan Song <[email protected]>
|
Does this PR depend on nydusd and nydus-image 2.2+ ? |
changweige
reviewed
Feb 16, 2023
| // stream into nydus blob directly, the tar2rafs eliminates the | ||
| // need to decompress it to a local directory first, thus greatly | ||
| // accelerating the pack process. | ||
| FeatureTar2Rafs Feature = "--type tar-rafs" |
| @@ -392,61 +405,86 @@ func packRef(ctx context.Context, dest io.Writer, opt PackOption) (io.WriteClose | |||
| } | |||
| }() | |||
|
|
|||
| blobMetaPath := filepath.Join(workDir, "blob.meta") | |||
| blobMetaFifo, err := fifo.OpenFifo(ctx, blobMetaPath, syscall.O_CREAT|syscall.O_RDONLY|syscall.O_NONBLOCK, 0644) | |||
| rafsBlobPath := filepath.Join(workDir, "blob.rafs") | |||
There was a problem hiding this comment.
Make "blob.rafs" as a constant string ?
There was a problem hiding this comment.
It's a temp file, just be used as a unique name.
| if err != nil { | ||
| return nil, errors.Wrapf(err, "create fifo file") | ||
| } | ||
|
|
||
| sourcePath := filepath.Join(workDir, "blob.targz") | ||
| sourceFifo, err := fifo.OpenFifo(ctx, sourcePath, syscall.O_CREAT|syscall.O_WRONLY|syscall.O_NONBLOCK, 0644) | ||
| tarBlobPath := filepath.Join(workDir, "blob.targz") |
There was a problem hiding this comment.
Make "blob.targz" as a constant string ?
changweige
reviewed
Feb 16, 2023
Support the new option `--type tar-rafs` of builder for Pack method in converter package, enables converting OCI tar blob stream into nydus blob directly, eliminating the need to decompress it to a local directory first, thus greatly accelerating the pack process. Signed-off-by: Yan Song <[email protected]>
The Pack conversion workflow will hang on `io.Copy` once the builder execution failed, this patch fixes it. Signed-off-by: Yan Song <[email protected]>
The `Features` option enables feature collection for nydus-image builder. Signed-off-by: Yan Song <[email protected]>
Signed-off-by: Yan Song <[email protected]>
We should fallback some cli flags automatically for older versions of builder. Signed-off-by: Yan Song <[email protected]>
The `PrefetchPatterns` option should be paased to the builder (nydus-image) merge subcommand, otherwise the prefetch feature table will not work in the final bootstrap (used in nydus image). Signed-off-by: Yan Song <[email protected]>
Signed-off-by: Yan Song <[email protected]>
imeoer
force-pushed
the
converter-speedup
branch
from
63d75c9 to
7850fe7
Compare
Signed-off-by: Yan Song <[email protected]>
imeoer
added a commit
to imeoer/acceleration-service
that referenced
this pull request
Mar 1, 2023
See containerd/nydus-snapshotter#352 It's a internal optimization, so just only bump nydus-snapshotter package to latest version. Signed-off-by: Yan Song <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Yan Song [email protected]