-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: frequent test timeouts on Android builders #42212
Comments
2022-04-17T00:58:13-33da011-e948c33/android-386-emu |
It looks like that tests are blocked forever at a syscall (sleep, park, etc.) Looking at the timeline, it feels like an issue with the emulator itself. Hence I reported to Corellium, but it would not be much helpful for them to tackle down the problem, I think. Is there anything we could really do? Perhaps mark this as unfortunate? |
I can stop triaging failures on this builder, but at that point the port becomes basically abandoned unless somebody else starts triaging it. 🤷♂️ We get a high enough rate of these kinds of failures that they should be reproducible with enough iterations, or at least could be caught with sufficient logging and a watchdog timeout. (Maybe somebody needs to sit down with the emulator and a remote debugger and figure out what's wedged?) |
|
|
Change https://go.dev/cl/412174 mentions this issue: |
Issue golang/go#42212 manifests as test timeouts, and is by far the most frequent of these known issues. Issue golang/go#51001 causes failures with "systemstack called from unexpected goroutine". It seems to have been introduced sometime last year, but it isn't clear to me whether it is a regression or an older (latent) bug unearthed by some other change. Issue golang/go#52724 appears to be a bug or race in the Android emulator itself. It might require a builder image update and/or escalation to the maintainers of the emulator proper. Updates golang/go#53377. Change-Id: I677915b1ff02dd02e0f14c63b0d25caf11e27a72 Reviewed-on: https://go-review.googlesource.com/c/build/+/412174 Reviewed-by: David Chase <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Run-TryBot: Bryan Mills <[email protected]>
Found new dashboard test flakes for:
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/http2/h2c.TestPropagation (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/http2/hpack (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/idna (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/internal/socket (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/internal/socks (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/internal/timeseries (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/nettest (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/netutil.TestLimitListenerClose (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@841c3eb1 x/net/proxy (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/http2/h2c.TestPropagation (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/http2/hpack (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/internal/socket (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/internal/socks.TestDial (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/internal/sockstest (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/internal/timeseries (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/ipv4 (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/netutil (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@f721fa3b x/net/proxy (log)
|
Found new dashboard test flakes for:
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/http2/h2c.TestPropagation (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/http2/hpack (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/internal/socket (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/internal/socks.TestDial (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/internal/sockstest (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/internal/timeseries (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/ipv6.TestSetICMPFilter (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/nettest (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/netutil (log)
2023-01-05 16:42 android-amd64-emu net@f8411da7 go@0bbd67e5 x/net/proxy (log)
|
The above failures are all in the |
Found new dashboard test flakes for:
2023-01-04 15:40 android-386-emu term@1efcd90d go@f53137f9 x/term (log)
|
Change https://go.dev/cl/465156 mentions this issue: |
I had initially added known issues fairly aggressively in order to use them to reduce noise in 'greplogs -triage'. Now that we are using 'watchflakes' for triage, that noise reduction is no longer important (the failures are already clustered to their respective known issues), and having greyed-out cells on the dashboard makes new regressions too easy to miss. Concretely: - golang/go#42212 is mostly specific to x/net at this point (as golang/go#57841) - There have been no failures matching golang/go#51001 since October. - golang/go#52724 has been so rare lately that we hadn't yet added a 'watchflakes' pattern for it. - There have been no failures matching golang/go#51443 since May. - There have been no failures matching golang/go#53116 or golang/go#53093 since I enabled 'watchflakes' for the builder in December. - The linux-amd64-perf builder seems to be passing consistently for x/benchmarks and x/tools, so there is no need to refer to golang/go#53538 to explain failures on it. Change-Id: Ia16db2a23e5fa037a299f1f56fb26f1cf84521e1 Reviewed-on: https://go-review.googlesource.com/c/build/+/465156 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Auto-Submit: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
Found new dashboard test flakes for:
2023-02-08 21:58 android-amd64-emu oauth2@adbaf66a go@01a5a83c x/oauth2 (log)
2023-02-08 21:58 android-amd64-emu oauth2@adbaf66a go@01a5a83c x/oauth2/authhandler (log)
2023-02-08 21:58 android-amd64-emu oauth2@adbaf66a go@01a5a83c x/oauth2/clientcredentials (log)
2023-02-08 21:58 android-amd64-emu oauth2@adbaf66a go@01a5a83c x/oauth2/endpoints (log)
2023-02-08 21:58 android-amd64-emu oauth2@adbaf66a go@01a5a83c x/oauth2/google (log)
|
Found new dashboard test flakes for:
2023-02-07 17:11 android-386-emu term@d974fe83 go@52fea4b3 x/term (log)
|
Found new dashboard test flakes for:
2023-02-07 17:11 android-386-emu term@d974fe83 go@d20e688f x/term (log)
|
Found new dashboard test flakes for:
2023-04-04 18:21 android-amd64-emu term@7ae6be6d go@1444c0b0 x/term (log)
|
Found new dashboard test flakes for:
2023-05-04 15:43 android-amd64-emu term@119f7033 go@a094a824 x/term (log)
|
Found new dashboard test flakes for:
2023-05-23 19:43 android-386-emu sys@b5c7a097 go@3824d3d7 x/sys/cpu (log)
2023-05-23 19:43 android-386-emu sys@b5c7a097 go@3824d3d7 x/sys/execabs (log)
2023-05-23 19:43 android-386-emu sys@b5c7a097 go@3824d3d7 x/sys/internal/unsafeheader (log)
|
Found new dashboard test flakes for:
2023-06-06 21:43 android-386-emu go@3ba9c890 runtime.TestParallelRWMutexReaders (log)
|
Found new dashboard test flakes for:
2023-09-12 16:41 android-amd64-emu sys@fdc7ef40 go@5cc1e99f x/sys/cpu (log)
2023-09-12 16:41 android-amd64-emu sys@fdc7ef40 go@5cc1e99f x/sys/execabs (log)
2023-09-12 16:41 android-amd64-emu sys@fdc7ef40 go@5cc1e99f x/sys/unix (log)
|
Found new dashboard test flakes for:
2023-10-11 14:06 android-386-emu term@6a610bc5 go@d4863909 x/term (log)
|
Found new dashboard test flakes for:
2023-10-11 14:06 android-386-emu time@b24d3b5e go@289b823a x/time/rate (log)
|
Found new dashboard test flakes for:
2023-11-08 17:59 android-386-emu sys@e4099bfa go@e1dc209b x/sys/cpu (log)
2023-11-08 17:59 android-386-emu sys@e4099bfa go@e1dc209b x/sys/execabs (log)
2023-11-08 17:59 android-386-emu sys@e4099bfa go@e1dc209b x/sys/unix (log)
|
Found new dashboard test flakes for:
2023-11-07 19:02 android-amd64-emu term@70d3a0bd go@eda42f7c x/term (log)
|
Found new dashboard test flakes for:
2023-10-11 14:06 android-386-emu time@b24d3b5e go@4346ba34 x/time/rate (log)
|
Found new dashboard test flakes for:
2023-11-07 19:02 android-amd64-emu term@70d3a0bd go@6b43a172 x/term (log)
|
Found new dashboard test flakes for:
2023-11-07 19:02 android-386-emu term@70d3a0bd go@e41fb0fd x/term (log)
|
Found new dashboard test flakes for:
2024-01-03 06:57 android-amd64-emu sys@0829ab15 go@ead47b0a x/sys/cpu (log)
2024-01-03 06:57 android-amd64-emu sys@0829ab15 go@ead47b0a x/sys/execabs (log)
2024-01-03 06:57 android-amd64-emu sys@0829ab15 go@ead47b0a x/sys/unix (log)
|
Found new dashboard test flakes for:
2023-11-21 18:52 android-amd64-emu time@883aed51 go@a95136a8 x/time/rate (log)
|
Found new dashboard test flakes for:
2024-01-04 15:06 android-amd64-emu term@ae941452 go@5a61d8d3 x/term (log)
|
Found new dashboard test flakes for:
2024-02-21 20:50 android-amd64-emu pkgsite@463d7c94 go@05c05796 x/pkgsite/cmd/internal/pkgsite (log)
2024-02-21 20:50 android-amd64-emu pkgsite@463d7c94 go@05c05796 x/pkgsite/internal (log)
2024-02-21 20:50 android-amd64-emu pkgsite@463d7c94 go@05c05796 x/pkgsite/internal/cache (log)
|
Found new dashboard test flakes for:
2024-02-21 20:50 android-386-emu pkgsite@463d7c94 go@7fd62ba8 x/pkgsite/internal (log)
2024-02-21 20:50 android-386-emu pkgsite@463d7c94 go@7fd62ba8 x/pkgsite/internal/cache (log)
|
Found new dashboard test flakes for:
2024-02-21 20:50 android-386-emu pkgsite@463d7c94 go@f278f756 x/pkgsite/internal/cache (log)
2024-02-21 20:50 android-386-emu pkgsite@463d7c94 go@f278f756 x/pkgsite/internal/config/serverconfig (log)
|
Found new dashboard test flakes for:
2024-02-21 20:50 android-386-emu pkgsite@463d7c94 go@d892cb49 x/pkgsite/internal/cache (log)
2024-02-21 20:50 android-386-emu pkgsite@463d7c94 go@d892cb49 x/pkgsite/internal/config/serverconfig (log)
2024-02-21 20:50 android-amd64-emu pkgsite@463d7c94 go@d892cb49 x/pkgsite/internal/fetch (log)
2024-02-21 20:50 android-amd64-emu pkgsite@463d7c94 go@d892cb49 x/pkgsite/internal/fetchdatasource (log)
2024-02-21 20:50 android-amd64-emu pkgsite@463d7c94 go@d892cb49 x/pkgsite/internal/frontend (log)
|
Found new dashboard test flakes for:
2024-04-04 14:40 android-amd64-emu sys@cabba82f go@2e064cf1 x/sys/cpu (log)
2024-04-04 14:40 android-amd64-emu sys@cabba82f go@2e064cf1 x/sys/execabs (log)
2024-04-04 14:40 android-amd64-emu sys@cabba82f go@2e064cf1 x/sys/unix (log)
|
Found new dashboard test flakes for:
2024-04-16 17:16 android-amd64-emu arch@5d0b1102 go@deeebf56 x/arch/arm/armasm (log)
2024-04-16 17:16 android-amd64-emu arch@5d0b1102 go@deeebf56 x/arch/arm64/arm64asm (log)
2024-04-16 17:16 android-amd64-emu arch@5d0b1102 go@deeebf56 x/arch/ppc64/ppc64asm (log)
|
2020-10-24T13:24:49-ef9fd89/android-amd64-emu
2020-10-18T23:04:17-eeed37f/android-386-emu
2020-10-18T23:04:17-eeed37f/android-amd64-emu
2020-10-15T00:08:50-e3ed001/android-amd64-emu
2020-10-07T16:58:08-a893ed3/android-386-emu
2020-09-26T10:08:07-9d91bd6/android-386-emu
2020-08-24T13:15:25-c12d262/android-amd64-emu
2020-08-17T08:59:35-3ff754b/android-386-emu
2020-08-03T21:05:38-64077c9/android-amd64-emu
2020-06-15T20:00:32-f1bc736/android-amd64-emu
2020-06-15T19:00:26-2780627/android-amd64-emu
2020-05-11T23:29:37-7e40ca2/android-386-emu
2020-04-28T20:04:54-593003d/android-386-emu
2020-04-28T20:04:54-593003d/android-amd64-emu
2020-04-27T17:57:16-29b5707/android-386-emu
2020-04-27T17:57:16-29b5707/android-amd64-emu
2020-04-20T16:35:11-1957bb5/android-amd64-emu
2020-03-01T15:39:31-2f85c7e/android-amd64-emu
2020-01-24T20:44:21-9fbb57f/android-386-emu
2020-01-13T16:29:24-86b9105/android-386-emu
2020-01-13T16:29:24-86b9105/android-amd64-emu
2020-01-03T14:33:44-a1369af/android-amd64-emu
2019-11-28T01:58:09-6d18c01/android-amd64-emu
2019-11-13T15:49:46-fee6b63/android-amd64-emu
2019-10-29T15:55:21-f43be2a/android-amd64-emu
2019-10-27T21:15:39-f8518d3/android-386-emu
2019-10-23T15:13:26-f89234f/android-386-emu
2019-10-22T10:09:44-742c48e/android-amd64-emu
CC @tklauser @hyangah @golang/release
The text was updated successfully, but these errors were encountered: