status updates for run_after_success jobs seem to be bugged

[BenTheElder]

Observed in the wild here:
kubernetes/kubernetes#51064 (comment)

The kubeadm job failed but was still marked as "Parent Job Status Changed: Job triggered."

/cc @luxas @Kargakis

[BenTheElder]

/cc @dims

[0xmichalis]

The kubeadm job was never meant to trigger for the linked PR (see run_if_changed below; fixed by #4586)

test-infra/prow/config.yaml

Lines 93 to 98 in b4a4bd5

	- name: pull-kubernetes-e2e-kubeadm-gce
	agent: kubernetes
	context: pull-kubernetes-e2e-kubeadm-gce
	max_concurrency: 8
	skip_report: true
	run_if_changed: '^(cmd/kubeadm|build/debs).*$'

Also, it was never meant to report (skip_report: true). Maybe #4563 was deployed in the meantime?

[krzyzacy]

hummm, @Kargakis we are still running 0.46

[0xmichalis]

Anything interesting in the plank log?

[krzyzacy]

nothing from plank
hook has bunch of
{"context":"pull-kubernetes-e2e-kubeadm-gce","id":1563938134,"level":"info","msg":"Status description Parent Job Status Changed: Job triggered..","org":"kubernetes","repo":"kubernetes","sha":"e0f11befdf1a5180678ba3853dc90c4a7de99164","state":"pending","time":"2017-09-20T19:27:22Z"}

[BenTheElder]

Hmm this still seems to be a problem: https://kubernetes.slack.com/archives/C09R23FHP/p1506049583000002

The kubemark job also seems to be consistently failing. 😕

Edit: #4684 is open for fixing the kubeadm tests themselves, but they should still be getting marked as failed (and apparently aren't?)

Edit 2: you can see this in the wild on one of our test PRs: kubernetes/kubernetes#46662
pull-kubernetes-e2e-kubeadm-gce — Parent Job Status Changed: Job triggered. even though on gubernator we can see that the job failed.

[BenTheElder]

/assign

[BenTheElder]

see also: #4564

[BenTheElder]

I think it may actually be possible that these are PRs that triggered pull-kubernetes-e2e-kubeadm-gce before it was changed to be run_if_changed. I'm testing this theory in kubernetes/kubernetes#52755

[0xmichalis]

Can you test with the latest version of plank? We are running it w/o a problem but we don't use run_after_success jobs yet. I can test it tomorrow.

[krzyzacy]

cc @cjwagner -
I think we can update plank tomorrow after 1.8 gets out

[BenTheElder]

@Kargakis Unfortunately the bazel build is now failing on that dummy PR so it's not getting tested with anything yet while I look into that 😞

[BenTheElder]

Oh hmm @luxas pull-kubernetes-e2e-kubeadm-gce is set to skip_report: true now. Should that still be the case?

[BenTheElder]

It doesn't seem to be reporting on PRs where it shouldn't anymore so I think we can close this now. I think this was just another issue with job configurations being in flight while PRs were open. @Kargakis if you do test this and find a problem re-open, but I think we can close this now. I can't find this in the wild anymore.

@luxas now that it only runs on build changes, should we flip skip_report back off?

[luxas]

Great, now it doesn't report on unrelated PRs, thanks!
(example: https://k8s-gubernator.appspot.com/pr/53119)
But it now does on https://k8s-gubernator.appspot.com/pr/53114, which touched cmd/kubeadm.

Whohoo, I can now flip skip_report: false 🎉 🎉

[BenTheElder]

This is still possibly an issue, I think it's actually because of stale PRs, re-opening while I look around. If it's just stale PRs I'll migrate to a new issue about that.

[BenTheElder]

/close
this is a broader issue, see #4912