Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

twoliter: fetch command for downloading and extracting external kits #270

Merged
merged 1 commit into from
Jun 10, 2024
Merged

twoliter: fetch command for downloading and extracting external kits #270

merged 1 commit into from
Jun 10, 2024

Conversation

jmt-lab
Copy link
Contributor

@jmt-lab jmt-lab commented Jun 4, 2024

Issue number: 258

Closes #258

Description of changes:

  • Adds functionality to Lock to download and extract kits resolved by Twoliter.lock to build/external-kits
  • Writes a metadata file that can be used by rpm2kit for generating the kit metadata for publishing a local kit
  • Adds twoliter fetch command
  • Remove sdk from remaining in lock resolution as an sdk image is not a kit image and does not have metadata

Testing done:

  • Uploaded a test kit image to ECR and updated Twoliter.toml to depend on it
  • Ran twoliter update to resolve Twoliter.lock
  • Ran twoliter fetch to fetch kits
  • Verified metadata and extraction path

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@jmt-lab jmt-lab self-assigned this Jun 4, 2024
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from 9827e92 to ccfedc2 Compare June 4, 2024 23:01
webern
webern reviewed Jun 4, 2024
View reviewed changes
twoliter/src/cmd/fetch.rs Outdated Show resolved Hide resolved
twoliter/src/cmd/fetch.rs
use std::path::PathBuf;

#[derive(Debug, Parser)]
pub(crate) struct Fetch {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I always ask this. Would we anticipate further subcommands? twoliter fetch sdk and twoliter fetch sources for example? If so, should this be twoliter fetch kits?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave this one unresolved since it's an outstanding question with no corresponding change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah hit resolve by accident there, I'm not sure either way, I kind of like it being just fetch cause it operates off Twoliter.lock which is generated just by update

twoliter/src/main.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
cbgbt
cbgbt reviewed Jun 5, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
cbgbt
cbgbt reviewed Jun 5, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
cbgbt
cbgbt reviewed Jun 5, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
sam-berning
sam-berning reviewed Jun 5, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from ccfedc2 to 93b18a1 Compare June 5, 2024 21:21
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 5, 2024

Updated by comments
Added digest to external kit metadata file

@jmt-lab jmt-lab requested review from webern, cbgbt and sam-berning June 5, 2024 21:21
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from 93b18a1 to bcdc898 Compare June 5, 2024 21:35
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 5, 2024

Fix clippy issues

cbgbt
cbgbt reviewed Jun 5, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch 2 times, most recently from f9eb662 to 91b2b57 Compare June 6, 2024 20:12
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 6, 2024

Added digest to LockedImage
Calculating digest for image list off the image index
Ensured external-kit-metadata.json is printed out in canonical json
Added external kit metadata to rerun-if-changed

webern
webern approved these changes Jun 6, 2024
View reviewed changes
tools/buildsys/src/main.rs Outdated Show resolved Hide resolved
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from 91b2b57 to e94f8b6 Compare June 6, 2024 20:25
cbgbt
cbgbt reviewed Jun 6, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch 2 times, most recently from afc311f to 7988930 Compare June 6, 2024 21:06
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 6, 2024

Updated const for metadata

bcressey
bcressey reviewed Jun 6, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
tools/buildsys/src/main.rs Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Show resolved Hide resolved
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from 7988930 to 4eda74f Compare June 6, 2024 22:27
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 6, 2024

Updated per ben's comments

@jmt-lab jmt-lab requested a review from bcressey June 6, 2024 22:29
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from 4eda74f to 799cd5a Compare June 6, 2024 22:37
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 6, 2024

Forgot to add license field to new crate

bcressey
bcressey reviewed Jun 7, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from 799cd5a to d90d486 Compare June 7, 2024 18:21
@jmt-lab jmt-lab requested a review from bcressey June 7, 2024 18:26
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 7, 2024

Updates to comments including

  • Add ValidIdentifier to validate the name and vendor fields
  • Implemented caching of the oci archive and external kits
  • Added buildsys changes here so clean-kits will clean up external-kit caches as well

@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from d90d486 to 1b21d9b Compare June 7, 2024 20:21
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 7, 2024

Fixed tests

@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from 1b21d9b to 8c62a22 Compare June 7, 2024 20:40
bcressey
bcressey reviewed Jun 7, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
@jmt-lab
Copy link
Contributor Author

jmt-lab commented Jun 7, 2024

Updated per latest comments

@jmt-lab jmt-lab requested a review from bcressey June 7, 2024 23:21
@bcressey bcressey mentioned this pull request Jun 10, 2024
Closed
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch 2 times, most recently from ed37792 to b15ee84 Compare June 10, 2024 20:58
bcressey
bcressey reviewed Jun 10, 2024
View reviewed changes
twoliter/src/lock.rs Outdated Show resolved Hide resolved
twoliter/src/lock.rs Outdated Show resolved Hide resolved
@jmt-lab jmt-lab force-pushed the ootb/external-kits/download branch from b15ee84 to 9afedf2 Compare June 10, 2024 21:36
@jmt-lab jmt-lab merged commit 9f6f5ed into bottlerocket-os:develop Jun 10, 2024
1 check passed
@jmt-lab jmt-lab deleted the ootb/external-kits/download branch June 10, 2024 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cbgbt cbgbt cbgbt left review comments

@bcressey bcressey bcressey approved these changes

@webern webern webern approved these changes

@sam-berning sam-berning Awaiting requested review from sam-berning

Assignees

@jmt-lab jmt-lab

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

twoliter: download and extract kits to build folder
5 participants
@jmt-lab @bcressey @cbgbt @webern @sam-berning