Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: netbsd-amd64-8_0 builder timing out frequently #31858

Open
bcmills opened this issue May 6, 2019 · 8 comments
Open

x/build: netbsd-amd64-8_0 builder timing out frequently #31858

bcmills opened this issue May 6, 2019 · 8 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented May 6, 2019

Four of the last five failures on the netbsd-amd64-8_0 builder are timeouts in various tests.
I can't tell whether the builder is slow or the test is triggering some sort of deadlock.

https://build.golang.org/log/a64cd49d7d367f90e6b79ab84714b4c065763168
https://build.golang.org/log/f7f50b085698aa5480d2c34a3e6c250595317bf6
https://build.golang.org/log/7b749c8616a9b7e9a77c7735038673f9a9dc5d37
https://build.golang.org/log/26dd20510233ec659adf0d5987690164d8abaa6a

CC @bsiegert @bradfitz
@bcmills bcmills added Testing An issue that has been verified to require only test changes, not just a test failure. OS-NetBSD NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 6, 2019
@gopherbot gopherbot added this to the Unreleased milestone May 6, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label May 6, 2019
@bsiegert
Copy link
Contributor

bsiegert commented May 6, 2019

I think it's just slowness. Maybe increasing the timeout helps?

@bradfitz
Copy link
Contributor

bradfitz commented May 6, 2019

@bsiegert, what makes you think that? I suspect the NetBSD port (or NetBSD) is buggy based on looking at historical data over the past month on this builder. The cgo_test has a 43.2 second median over 309 runes. And the cmd/compile/internal/types test is fast across all builders: 3.4 seconds on linux-amd64 at the slowest, 34.1 seconds on linux-arm on physical ARM hardware. js-wasm does it in 8.2 seconds. Why can't netbsd-amd64 do it in under 20 minutes?

So, no, I don't think it's just slowness.

@zoulasc
Copy link
Contributor

zoulasc commented May 11, 2019

Makes sense (that the problem lies with NetBSD or the NetBSD port). Is there a crash dump of where it is stuck?

@bradfitz
Copy link
Contributor

@zoulasc, nope. Our build system just gives up on it and doesn't try to actively grab a stack. (which varies by OS, anyway) If you want to give me basic instructions for capturing some debug info we could probably wire it up to report that before failing.

@zoulasc
Copy link
Contributor

zoulasc commented May 17, 2019

I will setup a local build and try to reproduce it locally so I can give you something that works...

@bcmills bcmills closed this as completed Jun 26, 2019
@bcmills bcmills reopened this Jun 26, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Jun 26, 2019

Still an issue. misc/cgo/test, os/signal, and go/types all have recent timeouts:
https://build.golang.org/log/a1652930ea41da5e9588f973a4c8a91878c00062
https://build.golang.org/log/9fcd7b26fe2a11d9883c6c18c370e36e3b3039b0
https://build.golang.org/log/a511a477b595ee1e3f444ed2b3e943bbc9e65541
https://build.golang.org/log/c0e5926e39b4704f6ff9bcfb23e41de1237ce859

There is also a cmd/go timeout that may or may not be related:
https://build.golang.org/log/8438c6756c49694e123c5b058e2a7903ece9d7f1

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/185217 mentions this issue: env/netbsd-amd64: update to NetBSD 8.1

gopherbot pushed a commit to golang/build that referenced this issue Jul 8, 2019
@bsiegert
env/netbsd-amd64: update to NetBSD 8.1
0fd1ffd 
NetBSD 8.1 is a bugfix release on the stable (8) branch and has
a number of recommended updates and fixes.

Updates golang/go#31858

Change-Id: Id0f4a2180af1f84486aa6bc1c4406c5e8d5a0cb3
Reviewed-on: https://go-review.googlesource.com/c/build/+/185217
Reviewed-by: Andrew Bonventre <[email protected]>
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
@bsiegert @codebien
env/netbsd-amd64: update to NetBSD 8.1
ca0abbe 
NetBSD 8.1 is a bugfix release on the stable (8) branch and has
a number of recommended updates and fixes.

Updates golang/go#31858

Change-Id: Id0f4a2180af1f84486aa6bc1c4406c5e8d5a0cb3
Reviewed-on: https://go-review.googlesource.com/c/build/+/185217
Reviewed-by: Andrew Bonventre <[email protected]>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/226977 mentions this issue: dashboard, env/netbsd-{amd64,386}: add NetBSD 9.0 builders

gopherbot pushed a commit to golang/build that referenced this issue Apr 7, 2020
@dmitshur
dashboard, env/netbsd-{amd64,386}: add NetBSD 9.0 builders
219acbf 
Update NetBSD amd64 and 386 builders to the latest NetBSD 9.0 release,
using the latest versions of anita (2.3) and pkgsrc packages (2019Q4).

Also increase the image disk size from 4 GB to 16 GB. The motivation
is to help tests that require more disk space and large dependencies.
Additional adjustments to the image building script will need to be
made to make use of increased disk space for the /tmp volume, as it
is currently being mounted in RAM with size equal to 25% of memory.

Remove the manual invocation of the mozilla-rootcerts install command.
It no longer works as is in NetBSD 9.0. The mozilla-rootcerts-openssl
package is added instead, which successfully performs that same task.
Hopefully this will be a more robust approach for the future.

Continue to use an explicit URL for the NetBSD release download, and
remove unused find_latest_release.

The plan is to roll out these builders first (without trybots), gain
more confidence, enable trybots, and then remove NetBSD 8.0 builders.

For golang/go#37244.
For golang/go#30839.
For golang/go#31858.
For golang/go#31726.
For golang/go#38283.

Change-Id: Id41278eaebe64cfbcf7b1ab9bfd64e550ee0bbf1
Reviewed-on: https://go-review.googlesource.com/c/build/+/226977
Reviewed-by: Benny Siegert <[email protected]>
Reviewed-by: Alexander Rakoczy <[email protected]>
@gabyhelp gabyhelp mentioned this issue Jun 20, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@bradfitz @zoulasc @bsiegert @bcmills @gopherbot