Spurious Android testing failure: connection reset by peer #38710
Labels
A-spurious
Area: Spurious failures in builds (spuriously == for no apparent reason)
O-android
Operating system: Android
Comments
alexcrichton
added
O-android
alexcrichton
added a commit
to alexcrichton/rust
that referenced
this issue
Jan 6, 2017
Local testing showed that I was able to reproduce an error where debuginfo tests on Android would fail with "connection reset by peer". Further investigation turned out that the gdb tests are android with bit of process management: * First an `adb forward` command is run to ensure that the host's port 5039 is the same as the emulator's. * Next an `adb shell` command is run to execute the `gdbserver` executable inside the emulator. This gdb server will attach to port 5039 and listen for remote gdb debugging sessions. * Finally, we run `gdb` on the host (not in the emulator) and then connect to this gdb server to send it commands. The problem was happening when the host's gdb was failing to connect to the remote gdbserver running inside the emulator. The previous test for this was that after `adb shell` executed we'd sleep for a second and then attempt to make a TCP connection to port 5039. If successful we'd run gdb and on failure we'd sleep again. It turns out, however, that as soon as we've executed `adb forward` all TCP connections to 5039 will succeed. This means that we would only ever sleep for at most one second, and if this wasn't enough time we'd just fail later because we would assume that gdbserver had started but it may not have done so yet. This commit fixes these issues by removing the TCP connection to test if gdbserver is ready to go. Instead we read the stdout of the process and wait for it to print that it's listening at which point we start running gdb. I've found that locally at least I was unable to reproduce the failure after these changes. Closes rust-lang#38710
This was referenced Jan 6, 2017
bors
added a commit
that referenced
this issue
Jan 8, 2017
compiletest: Fix flaky Android gdb test runs Local testing showed that I was able to reproduce an error where debuginfo tests on Android would fail with "connection reset by peer". Further investigation turned out that the gdb tests are android with bit of process management: * First an `adb forward` command is run to ensure that the host's port 5039 is the same as the emulator's. * Next an `adb shell` command is run to execute the `gdbserver` executable inside the emulator. This gdb server will attach to port 5039 and listen for remote gdb debugging sessions. * Finally, we run `gdb` on the host (not in the emulator) and then connect to this gdb server to send it commands. The problem was happening when the host's gdb was failing to connect to the remote gdbserver running inside the emulator. The previous test for this was that after `adb shell` executed we'd sleep for a second and then attempt to make a TCP connection to port 5039. If successful we'd run gdb and on failure we'd sleep again. It turns out, however, that as soon as we've executed `adb forward` all TCP connections to 5039 will succeed. This means that we would only ever sleep for at most one second, and if this wasn't enough time we'd just fail later because we would assume that gdbserver had started but it may not have done so yet. This commit fixes these issues by removing the TCP connection to test if gdbserver is ready to go. Instead we read the stdout of the process and wait for it to print that it's listening at which point we start running gdb. I've found that locally at least I was unable to reproduce the failure after these changes. Closes #38710
frewsxcv
pushed a commit
to frewsxcv/rust
that referenced
this issue
Jan 9, 2017
Local testing showed that I was able to reproduce an error where debuginfo tests on Android would fail with "connection reset by peer". Further investigation turned out that the gdb tests are android with bit of process management: * First an `adb forward` command is run to ensure that the host's port 5039 is the same as the emulator's. * Next an `adb shell` command is run to execute the `gdbserver` executable inside the emulator. This gdb server will attach to port 5039 and listen for remote gdb debugging sessions. * Finally, we run `gdb` on the host (not in the emulator) and then connect to this gdb server to send it commands. The problem was happening when the host's gdb was failing to connect to the remote gdbserver running inside the emulator. The previous test for this was that after `adb shell` executed we'd sleep for a second and then attempt to make a TCP connection to port 5039. If successful we'd run gdb and on failure we'd sleep again. It turns out, however, that as soon as we've executed `adb forward` all TCP connections to 5039 will succeed. This means that we would only ever sleep for at most one second, and if this wasn't enough time we'd just fail later because we would assume that gdbserver had started but it may not have done so yet. This commit fixes these issues by removing the TCP connection to test if gdbserver is ready to go. Instead we read the stdout of the process and wait for it to print that it's listening at which point we start running gdb. I've found that locally at least I was unable to reproduce the failure after these changes. Closes rust-lang#38710
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
First seen as https://travis-ci.org/rust-lang/rust/jobs/187737907, I've also seen this from time to time on other locations but haven't ever tracked it down. Wanted to have an issue to track it though!
Failures tend to look like:
The text was updated successfully, but these errors were encountered: