Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce k0s airgap bundle-artifacts #5360

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

twz123
Copy link
Member

@twz123 twz123 commented Dec 13, 2024

Description

This sub-command introduces a "k0s native" way to create airgap artifact bundles. The current way to do this requires either a Docker or containerd daemon. Even then, there are certain scenarios where the created bundles don't contain all the necessary tags, i.e. when images are referenced by both their tag and their digest.

Replace image-bundler with k0s airgap bundle-artifacts. Dog-food the new sub-command in the k0s build itself.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

How Has This Been Tested?

  • Manual test
  • Auto test added

Checklist:

  • My code follows the style guidelines of this project
  • My commit messages are signed-off
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • I have checked my code and corrected any misspellings

Sorry, something went wrong.

@twz123 twz123 added enhancement New feature or request area/cli labels Dec 13, 2024
@twz123 twz123 marked this pull request as ready for review December 13, 2024 20:52
@twz123 twz123 requested review from a team as code owners December 13, 2024 20:52
Copy link
Contributor

This pull request has merge conflicts that need to be resolved.

return refs, nil
}

func parseArtifactRefsFromSeq(refs iter.Seq[string]) (collected []reference.Named, _ error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _ error kind of looks like the return value can be ignored.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I can follow. What would you suggest?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not using named returns, I suppose, they are terrible :)

Copy link
Member Author

@twz123 twz123 Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO named returns kinda come in handy on occasions like this. I removed them, but that meant I had to duplicate the var declaration ¯\_(ツ)_/¯

Copy link
Contributor

This pull request has merge conflicts that need to be resolved.

Copy link
Contributor

github-actions bot commented Jan 9, 2025

This pull request has merge conflicts that need to be resolved.

Copy link
Contributor

github-actions bot commented Jan 9, 2025

This pull request has merge conflicts that need to be resolved.

Copy link
Contributor

This pull request has merge conflicts that need to be resolved.

Copy link
Contributor

This pull request has merge conflicts that need to be resolved.

For example, to create an image bundle that contains the images required for
the current configuration, use the following command:

k0s airgap list-images | k0s airgap bundle-artifacts -v -o image-bundle.tar
Copy link
Contributor

@makhov makhov Jan 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to run just k0s airgap bundle-artifacts -v -o image-bundle.tar without piping for the default images?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the idea having the input is for the users to be able to add their own images too. But that said, it could work also in the way that if the list is not given, then it could use the default images. But would that need config loading etc.?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's what I mean, if the input list is not provided, use default images. If I remember correctly, existing k0s airgap list-images reads config already, so the user could use pipes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that was kinda the idea: Not to add any config loading here at all and use pipes. Otherwise this would be duplicating quite a bit of the logic in list-images. This is also advertised in the long help text of k0s airgap:

For example, to create an image bundle that contains the images required for
the current configuration, use the following command:

k0s airgap list-images | k0s airgap bundle-artifacts -v -o image-bundle.tar

func parseArtifactRefsFromReader(in io.Reader) ([]reference.Named, error) {
words := bufio.NewScanner(in)
words.Split(bufio.ScanWords)
refs, err := parseArtifactRefsFromSeq(func(yield func(string) bool) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use good old for loop instead of generators here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm usually not a fan of these and this is no exception (seems unnecessarily complex) but given the deadline I'm OK merging this...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

type insecureRegistryFlag airgap.InsecureOCIRegistryKind

var insecureRegistryFlagValues = [...]string{
airgap.NoInsecureOCIRegistry: "no",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use just constants like

const (
   NoInsecureOCIRegistry = "no"
   SkipTLSVerifyOCIRegistry = "skip-tls-verify"
   PlainHTTPOCIRegistry = "plain-http"
)

Instead of the magic below?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could go for string constants, but that would only make sense if airgap.InsecureOCIRegistryKind was also stringly typed, and if the constants lived in pkg/airgap.

What I like about having this separate and number typed is that the current zero value is actually a good default (NoInsecureOCIRegistry). To get the same effect using strings, NoInsecureOCIRegistry would have to be the empty string, which doesn't feel right. In turn, not defining the empty string as a constant would mean that the zero value would live in an "undefined limbo", i.e. it wouldn't be valid. Also, the string value is only important for the command line flag, not for the actual implementation in pkg/airgap, so why not scope it to the command? We would need a way to validate raw input strings and provide a list of all available strings for command-line flag parsing anyway, so the code below wouldn't really go away unless we hard-coded it. But since it's just in this file, and there are probably no more options to come, that's fine.

return errors.New("must be one of " + strings.Join(insecureRegistryFlagValues[:], ", "))
}

type platformFlag imagespecv1.Platform
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, stupid question, what is this for?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a newtype over the OCI image spec's Platform type that implements pflag.Value so that it can be added as a flag: cmd.Flags().Var((*platformFlag)(&platform), "platform", "the platform to export")

RewriteTarget RewriteRefFunc
}

func (b *OCIArtifactsBundler) Run(ctx context.Context, refs []reference.Named, out io.Writer) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be probably split into a few functions but if it works I don't want to block the PR given the date.

This code is fairly complex calling loads of dependencies, is it mostly copied from an example or something?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be probably split into a few functions

I think the most interesting part to split up would be the loop body, which I tried to do. Added a function for the call to oras.Copy, and a method for creating the manifests for the copied ref.

is it mostly copied from an example or something?

Not that I remember. It was more like tweaking the call to oras.Copy and the contents of the archive index long enough to get the desired outcome.

return err
}

newSource := func(ref reference.Named) oras.ReadOnlyTarget {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one example, this should probably a separate function.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've moved it into the for loop, as this is basically just a struct literal. I think it's fine there after having factored out other functions/methods.

}
}

copyOpts := oras.CopyOptions{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO also worth extracting it to a separate function

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine, as this is just a struct literal? Converting this into a function with a flat argument list would somehow feel wrong to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair enough

@twz123 twz123 force-pushed the bundle-artifacts-cmd branch from 749e4dd to c9265d1 Compare January 23, 2025 16:33
This sub-command introduces a "k0s native" way to create airgap
artifact bundles. The current way to do this requires either a Docker
or containerd daemon. Even then, there are certain scenarios where the
created bundles don't contain all the necessary tags, i.e. when images
are referenced by both their tag and their digest.

Signed-off-by: Tom Wieczorek <[email protected]>
Dog-food the new sub-command in the k0s build itself.

Signed-off-by: Tom Wieczorek <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cli enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants