-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure kaniko e2e test pushes image to registry #158
Conversation
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here (e.g. What to do if you already signed the CLAIndividual signers
Corporate signers
|
I signed it! |
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. |
CLAs look good, thanks! |
Super excited to see this @tanner-bruce ! I'd like to give this a thorough review, you should see it shortly! In the meantime, I can see you added secrets for running tests locally, I assume so that you can push to your registry? I have two follow up requests/questions:
Thanks again for this contribution!!! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added my feedback! Thanks again for this @tanner-bruce :D
/ok-to-test whoa i didnt even know we needed that, i should probably add something to CONTRIBUTING.md 🤔 |
Everything should be addressed now @bobcatfish ! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A couple minor comments, one major comment re. I'm shocked that this works hahaha, but I think this is almost ready to go!
/meow space
test/kaniko_task_test.go
Outdated
if err != nil { | ||
t.Fatalf("Failed to read build-step-kaniko log stream: %v", err) | ||
} | ||
initLogs := string(bs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have 2 more requests:
- Can we move all the logic around getting the init logs into a separate function as well? My ideal would be that this part of the test looked something like this:
// The Build created by the TaskRun will have the same name
b, err := c.BuildClient.Get(kanikoTaskRunName, metav1.GetOptions{})
if err != nil {
t.Errorf("Expected there to be a Build with the same name as TaskRun %s but got error: %s", kanikoTaskRunName, err)
}
podLogs := getPodLogs(b...)
if !strings.Contains(podLogs, kanikoBuildOutput) {
t.Fatalf("Expected output %s from pod %s but got %s", kanikoBuildOutput, podName, string(bs))
}
initLogs := getInitLogs(b...)
if checkForDigest(initLogs,....) {
t.Fatalf("Expected latest digest to be __ but was __ ".....)
}
-
It is crazy that this is working!!! XD I'm running it repeatedly locally and it's succeeding but this goes against everything we've been seeing in Add log verification to integration tests #119 and Create solution for making logs available when running Tasks #143 - @tejal29 and I looked at it and if we switch this back to the hello world task, the init container logs are always gone when we look at them (we get an error like
Unable to retrieve container logs for docker://6489e33a7432f735953b542d910ecd6ab7033e13d075fa13b16ad3f3babcb052
), so it seems like the fact that this Build is going more is making it stick around longer, but we don't know how long!! I really don't want to have flakes in our tests so I think we have 2 options:- Commit the test as is, if we start seeing flakes, make the check optional
- Log a warning if we can't get the initLogs but don't fail the test <-- the downside here is that i'm pretty sure our test scripts will swallow any test output and we'd never see this, unless it happened locally
I'm actually leaning toward (i), since adding this is an improvement on what we had before (and once we finish #143 we can swap out the part that grabs the logs here), but maybe @tejal29 @aaron-prindle @imjasonh @dlorenc have a different opinion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that i think, i like 1. Or else, we might ignore the warning and its easier to not fix it.
Its easy to make the check optional later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-
For sure, I should have done this in the first place. I took a slightly different approach but I think you'll like it.
-
I agree, I was worried about flakiness when I started working on it. One thought I had that could be investigated is that
GetLogs
allows following the logs of a container. I think with that, as soon as the pod is created we could grab a handle to all of the containers log streams and build up the logs as they run. There could be a small race condition here but I think it would still be less than what could happen with the current behaviour.
With how I handled 1 we could easily test out my idea on 2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think with that, as soon as the pod is created we could grab a handle to all of the containers log streams and build up the logs as they run.
oooh that sounds like an interesting approach, i like it! I'm gonna lgtm this PR and put a hold on it, if you want to make this change as part of this PR feel free, otherwise feel free to remove the hold and we can go ahead with this, and address any flakes as they happen
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Adds `github.com/google/go-containerregistry` as a dependency for accessing container registries Adds `KANIKO_SECRET_CONFIG_FILE` for adding a service account to the kaniko task when running e2e test locally to ensure kaniko is able to push to a gcr.io registry.. This is necessary for running the e2e tests locally unless the kubernetes nodes provide another way of authenticating to the registry, such as provisioning them with a storage admin scope. Once tektoncd#151 is in place, we will be able to update this code to create a service account and use that in the BuildSpec. Fixes tektoncd#150
test/kaniko_task_test.go
Outdated
if err != nil { | ||
t.Fatalf("Failed to read build-step-kaniko log stream: %v", err) | ||
} | ||
initLogs := string(bs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think with that, as soon as the pod is created we could grab a handle to all of the containers log streams and build up the logs as they run.
oooh that sounds like an interesting approach, i like it! I'm gonna lgtm this PR and put a hold on it, if you want to make this change as part of this PR feel free, otherwise feel free to remove the hold and we can go ahead with this, and address any flakes as they happen
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great!!!! Thanks for iterating on this also :D
/lgtm
If you want to make any other changes go ahead, otherwise comment with /hold cancel
on its own line to merge as is:
/hold
No problem! /hold cancel |
whoops, forgot: |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bobcatfish, tanner-bruce The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Improves Release Process Documentation
Adds
github.com/google/go-containerregistry
as a dependency foraccessing container registries
Adds
KANIKO_SECRET_CONFIG_FILE
for adding a service account to thekaniko task when running e2e test locally.
Fixes #150