x/build: add NetBSD 9.0 builder

[bradfitz]

NetBSD 9.0 is out:
https://netbsd.org/releases/formal-9/NetBSD-9.0.html

Tracking bug for adding a builder.

/cc @bsiegert

[bsiegert]

You beat me to it :) Should it be in addition to the 8.0 builder, or instead of it? -- Benny

[toothrot]

I think we should replace the 8.0 builder with 9.0. NetBSD is not a first-class port, so I don't think we should expand our coverage to older versions unless there is some compelling reason.

[bradfitz]

NetBSD is not a first-class port, so I don't think we should expand our coverage to older versions unless there is some compelling reason.

Works for me.

[gopherbot]

Change https://golang.org/cl/226977 mentions this issue: dashboard, env/netbsd-{amd64,386}: update to NetBSD 9.0

[dmitshur]

(Our current netbsd-amd64-8_0 builder is running NetBSD 8.0RC1 with an image built 2 years ago in CL 109035. CL 185217 updated the env script to NetBSD 8.1, but a new image was not built. The netbsd-386-8_0 builder is disabled due to #31726.)

I agree with @toothrot that there isn't a need for 8.0 once 9.0 is working. However, I think it would be easier to add a new 9.0 builder, letting it run for some time to make sure it works better than 8.0, then remove the 8.0 builder.

We should also increase the disk size while building a new image. Its current one is very small (4 GB) and causes issues like #30839, and inability to install large dependencies needed for some tests. /cc @randall77

[dmitshur]

I've investigated what it would take to get a 9.0 image built in CL 226977. The image was built and boots up, buildlet runs, but it seems something went wrong with installing CA certificates on it. It fails when attempting to fetch an HTTPS URL:

golang.org/x/build $ go run ./cmd/debugnewvm -run-build=netbsd-amd64-8_0 -override-image=netbsd-amd64-9-0-2019q4
2020/04/02 12:09:00 Creating debug-temp-1585829340 (with VM image "netbsd-amd64-9-0-2019q4")
2020/04/02 12:09:02 instance requested
2020/04/02 12:09:50 instance created
2020/04/02 12:09:50 serial output error: googleapi: got HTTP response code 404 with body: Not Found
2020/04/02 12:09:50 got instance info; running in https://www.googleapis.com/compute/v1/projects/symbolic-datum-552/zones/us-central1-b
2020/04/02 12:09:50 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:09:55 client buildlet probe error: Get "http://10.240.0.2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/04/02 12:09:56 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:01 client buildlet probe error: Get "http://10.240.0.2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/04/02 12:10:02 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:07 client buildlet probe error: Get "http://10.240.0.2": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/04/02 12:10:08 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:08 client buildlet probe error: Get "http://10.240.0.2": dial tcp 10.240.0.2:80: connect: connection refused
2020/04/02 12:10:09 About to hit http://10.240.0.2 to see if buildlet is up yet...
2020/04/02 12:10:09 buildlet probe: 200 OK
2020/04/02 12:10:09 WorkDir: /tmp/workdir, <nil>
2020/04/02 12:10:09 Pushing 'go1.4' Go bootstrap dir ...
2020/04/02 12:10:09 Putting Go bootstrap: 500 Internal Server Error; body: fetching URL https://storage.googleapis.com/go-builder-data/gobootstrap-netbsd-amd64-2da6b33.tar.gz: Get https://storage.googleapis.com/go-builder-data/gobootstrap-netbsd-amd64-2da6b33.tar.gz: x509: certificate signed by unknown authority
exit status 1

In contrast, using the current netbsd-amd64-8-0-2018q1 image results in:

golang.org/x/build $ go run ./cmd/debugnewvm -run-build=netbsd-amd64-8_0
...
2020/04/02 12:14:31 About to hit http://10.240.0.5 to see if buildlet is up yet...
2020/04/02 12:14:31 buildlet probe: 200 OK
2020/04/02 12:14:31 WorkDir: /tmp/workdir, <nil>
2020/04/02 12:14:31 Pushing 'go1.4' Go bootstrap dir ...
2020/04/02 12:14:34 Pushing 'go' dir...
2020/04/02 12:14:42 Running src/all.bash ...
...

@bsiegert Do you have ideas what would cause that? Are you able to help with investigating this? Thanks.

[iamleot]

Hello Dmitri, Dmitri Shuralyov writes:

@bsiegert Do you have ideas what would cause that? Are you able to help with investigating this? Thanks.

Probably no rootcerts are found. Installing mozilla-rootcerts-openssl package (or manually populating them) will probably fix it.

[dmitshur]

@iamleot Note that the mozilla-rootcerts package is already being installed, and that was sufficient in NetBSD 8.0. See here and here. It seems likely something about the way it's being installed has changed and requires adjustment in the script.

But thanks for the suggestion, I can explore how the mozilla-rootcerts-openssl package differs.

[bsiegert]

You might need to add mozilla-rootcerts-openssl in addition to mozilla-rootcerts. I admit that I don't know why the latter alone is not enough.

[iamleot]

`mozilla-rootcerts' package installs certificates under /usr/pkg and - at least openssl - will not use them. In order to populate /etc/openssl with the certificates `mozilla-rootcerts install' (a script provided by mozilla-rootcerts package) should be manually invoked. `mozilla-rootcerts-openssl' package just automatically take care of also doing the last step.

[dmitshur]

@iamleot Thanks for providing context. However, note that the mozilla-rootcerts install script has been invoked manually on this line:

/usr/pkg/sbin/mozilla-rootcerts install

If adding the mozilla-rootcerts-openssl package fixes this problem, it's likely because that invocation above has stopped working. It seems more brittle to invoke that command via an absolute path than to install a package, so I'd be okay with removing the manual mozilla-rootcerts install invocation if it turns out to obsoleted by adding the mozilla-rootcerts-openssl package.

[dmitshur]

Adding the mozilla-rootcerts-openssl package has resolved the problem. It continued to work after I removed these two commands:

touch /etc/openssl/openssl.cnf
/usr/pkg/sbin/mozilla-rootcerts install

I'll go with that approach in the CL. Thanks for the suggestion.

[randall77]

FYI, I can allocate a netbsd 9.0 builder, but I can't do anything with it.

$ gomote create netbsd-amd64-9_0
# still creating netbsd-amd64-9_0 after 5s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 11s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 15s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 20s; 0 requests ahead of you
# still creating netbsd-amd64-9_0 after 26s; 0 requests ahead of you
user-khr-netbsd-amd64-9_0-0
$ gomote ssh user-khr-netbsd-amd64-9_0-0
2020/04/07 08:56:42 Builder type "netbsd-amd64-9_0" not known to this gomote binary. Update your gomote binary. TODO: teach gomote to fetch build configs from the server (Issue 30929)
$ go get -u golang.org/x/build/cmd/gomote
$ gomote ssh user-khr-netbsd-amd64-9_0-0
2020/04/07 09:03:07 Builder type "netbsd-amd64-9_0" not known to this gomote binary. Update your gomote binary. TODO: teach gomote to fetch build configs from the server (Issue 30929)

None of the other commands I tried (ls, push) work either.

[randall77]

In fact, I can't even destroy the one I just created.

[dmitshur]

@randall77 The NetBSD 9.0 builder is still in development, so you'll need to re-build gomote with CL 226977 applied to be able to use it at this time.

I'll update this thread when the new builder is operational.

[gopherbot]

Change https://golang.org/cl/227538 mentions this issue: dashboard, env/netbsd-{amd64,386}: mount /tmp on disk instead of RAM

[gopherbot]

Change https://golang.org/cl/227545 mentions this issue: dashboard: unset known issue for NetBSD 9.0 builders, use for trybots

[gopherbot]

Change https://golang.org/cl/227738 mentions this issue: dashboard: remove netbsd-{amd64,386}-8_0 builders