Make pull-kubernetes-e2e-gce-gpu blocking

[rohitagarwal003]

The pull-kubernetes-e2e-gce-gpu job has been running for a while on every PR and has been reasonably non-flaky: https://k8s-gubernator.appspot.com/builds/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-gpu

We recently stopped running all non-blocking tests on every PR in #4640.

I would like to make the pull-kubernetes-e2e-gce-gpu blocking if there are no objections.

[rohitagarwal003]

/assign @krzyzacy @BenTheElder @fejta

[rohitagarwal003]

/cc @vishh

[krzyzacy]

cc @spiffxp

[krzyzacy]

Let's make the gpu job working on prow instead of Jenkins and we can make it always run and blocking. @BenTheElder is working towards it.

[BenTheElder]

ref #4639 for the most recent PR, I've had a job pass now and once that config is updated I expect this should work. we can probably officially migrate to prow soon but I want to see it pass more :-)

[BenTheElder]

Following up, I've gotten one run in since #4639, and it was succesful.
Runs here, Sep 19 22:17:49 is the start time of the first one after that PR.
Also here under pull-kubernetes-e2e-gce-gpu-prow.

I will run more and keep an eye on them.

[BenTheElder]

This also works on release-1.7 and has 4 succesful runs since #4639.
I think we can probably switch the GPU job to run on Prow any time @mindprince.

[krzyzacy]

I think we can flip it to prow now to give @mindprince more signals :-)

[rohitagarwal003]

Awesome. Thanks guys!

[krzyzacy]

We'll watch the job for a few days (until code freeze maybe), and @mindprince you will want to propose to release team to make the job blocking (maybe join our burndown meeting next Monday?) :-)

[krzyzacy]

@jdumars FYI

[BenTheElder]

This job has been pretty well behaved, perhaps we can make it blocking after the release is done? I think we will need to propose it somewhere (the sig-testing meeting maybe?)

[spiffxp]

Just bumped into a quota issue

https://k8s-testgrid.appspot.com/kubernetes-presubmits#pull-kubernetes-e2e-gce-gpu

W0929 18:01:58.316] ERROR: (gcloud.compute.networks.create) Could not fetch resource:
W0929 18:01:58.317]  - Quota 'SUBNETWORKS' exceeded.  Limit: 160.0

[BenTheElder]

xref: #4472 for probably the best solution, though generally we should be fine via max_concurrency and the janitor.

[BenTheElder]

Adding a note: the networking quota issue has been solved for this job and the fix is being rolled out to other jobs (thanks @MrHohn !)

[krzyzacy]

/unassign @jdumars
/assign @mindprince @BenTheElder
I guess feel free to send or update the PR to make it blocking before 1.9

[BenTheElder]

@krzyzacy I did: #5134

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close