x/crypto/ssh: ssh: handshake failed: ssh: unexpected message type 3 (expected one of [6])

[lucaswxp]

What version of Go are you using (go version)?

go version go1.7.4 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/lucaswxp"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mw/y5nw62813y9105dl9n5wy6_40000gn/T/go-build710665517=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?

Tried creating a SSH client:

  c := &ssh.ClientConfig{User:"a",Auth:[]ssh.AuthMethod{ssh.Password("b")}}
  _, err := ssh.Dial("tcp", "x.x.x.x:22", c)

  if err != nil {
    fmt.Printf("%s", err)
 }

What did you expect to see?

Successful login. I can login in terminal, but any host I put in .Dial() I get the following error:

What did you see instead?

ssh: handshake failed: ssh: unexpected message type 3 (expected one of [6])

What is type 3 error? Any host I put I get this message.

[mblair]

I'm hitting this too. I've done a little bisecting, and it looks like this is the offending commit:

golang/crypto@a59c127

@hanwen

[hanwen]

const msgUnimplemented = 3
const msgServiceAccept = 6

what does the remote end run?

Can you run with debugHandshake and post the log output?

[hanwen]

oh, I can see this on tests against openssh. Investigating.

[zcalusic]

Hitting this bug, too. go1.8rc3 linux/amd64

Tnx for looking into it.

[tarys-db]

The bug was also present in 1.7.1.
I upgraded to 1.7.5 and it's still there.
To me it happens randomly (more often than not) when using "PublicKeys" auth.

[hanwen]

the Go version is probably irrelevant. This must be related to the recent changes in the SSH code.

[tarys-db]

Here is a debug output of a failed connection. Maybe that helps.

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 837
debug2: parse_server_config: config /etc/ssh/sshd_config len 837
debug1: sshd version OpenSSH_6.9, OpenSSL 1.0.2d 9 Jul 2015
debug1: private host key #0: ssh-rsa SHA256:---
debug1: private host key #1: ssh-dss SHA256::---
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:---
debug1: private host key #3: ssh-ed25519 SHA256::---
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-p'
debug1: rexec_argv[2]='22222'
debug1: rexec_argv[3]='-D'
debug1: rexec_argv[4]='-dd'
debug1: rexec_argv[5]='-e'
Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22222 on 0.0.0.0.
Server listening on 0.0.0.0 port 22222.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22222 on ::.
Server listening on :: port 22222.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug2: parse_server_config: config rexec len 837
debug1: sshd version OpenSSH_6.9, OpenSSL 1.0.2d 9 Jul 2015
debug1: private host key #0: ssh-rsa SHA256:---
debug1: private host key #1: ssh-dss SHA256:---
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:/---
debug1: private host key #3: ssh-ed25519 SHA256:/---
debug1: inetd sockets after dupping: 3, 3
Connection from 123.123.123.123 port 44694 on 172.31.1.100 port 22222
debug1: Client protocol version 2.0; client software version Go
debug1: no match: Go
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9p1 Ubuntu-2ubuntu0.2
debug2: fd 3 setting O_NONBLOCK
debug2: Network child is on pid 14539
debug1: permanently_set_uid: 107/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 [preauth]
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
debug2: kex_parse_kexinit: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: kex_parse_kexinit: none,[email protected] [preauth]
debug2: kex_parse_kexinit: none,[email protected] [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: first_kex_follows 0 [preauth]
debug2: kex_parse_kexinit: reserved 0 [preauth]
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
debug2: kex_parse_kexinit: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa,ssh-dss,ssh-ed25519 [preauth]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],arcfour256,arcfour128 [preauth]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],arcfour256,arcfour128 [preauth]
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha1,hmac-sha1-96 [preauth]
debug2: kex_parse_kexinit: hmac-sha2-256,hmac-sha1,hmac-sha1-96 [preauth]
debug2: kex_parse_kexinit: none [preauth]
debug2: kex_parse_kexinit: none [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: [preauth]
debug2: kex_parse_kexinit: first_kex_follows 0 [preauth]
debug2: kex_parse_kexinit: reserved 0 [preauth]
debug1: kex: client->server aes128-ctr hmac-sha2-256 none [preauth]
debug1: kex: server->client aes128-ctr hmac-sha2-256 none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug2: monitor_read: 6 used once, disabling now
debug2: set_newkeys: mode 1 [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug2: set_newkeys: mode 0 [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
dispatch_protocol_error: type 20 seq 3 [preauth]
Connection closed by 123.123.123.123 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: Killing privsep child 14539
debug1: audit_event: unhandled event 12
<

[hanwen]

can people try out https://go-review.googlesource.com/c/36055/1 to see if that fixes things?

[mblair]

@hanwen yep, that fixes things for me.

[stayclassychicago]

@hanwen ditto, fixes my issues too, consistent connections

[zcalusic]

Works fine here now, thank you @hanwen for your work!

[lucaswxp]

Hey guys, sorry, noob question here:
I should git clone the master and compile from source to get the patch?

Regards

[zcalusic]

I was perplexed for a while, so I see where you're coming from. :)

On the right side of https://go-review.googlesource.com/c/36055/1 there's a download button. It saves a complete junk, which you can decode with base64 -d command. After that go to your $GOPATH/golang.org/x/crypto folder and apply what you got from base64 utility with patch -p1 < ssh-patch.diff and finally recompile the utility which uses the library.

Yeah, too complicated IMHO, that gerrit tool not very user friendly, it seems.

[lucaswxp]

@zcalusic Ahh, that worked! Thanks, solved for me.

[eikenb]

I just ran into this to in the sftp package. It was triggering multiple integration tests to fail. I bisected it to the same commit (a59c127) and then found this ticket. I tried your commit and it fixed it for me as well.

[hanwen]

you can fetch the change by doing:

git fetch https://go.googlesource.com/crypto refs/changes/55/36055/1 && git checkout FETCH_HEAD

[CameronGo]

I've temporarily added @hanwen 's step above to my build script and that has solved the problem for me pending a permanent fix

[edmwagner]

This started occurring after a few runs successfully runs via my ide(intellij) and by hand. Found this and reviewed/applied the patch for local builds. Thus far, the issue looks to have been resolved.

I will update if I see the issue again today with the patch in place.

Thank you for finding and working on this so quickly.

[yobert]

I also hit this issue and the patch fixed it. Thanks!

If anyone wants--- I've got a temporary fork of x/crypto master with the patch applied, until it gets into master. Here -> https://github.com/yobert/crypto

[hanwen]

this is now submitted. Can y'all try out https://go.googlesource.com/crypto/+/77014cf7f9bde4925afeed52b7bf676d5f5b4285 and see if it fixes things for you?

[hiroakis]

I was hitting this. It works now after applying the patch.

[zcalusic]

Works fine here. 👍

[eikenb]

Fixed it for me. Thanks.

[unclejack]

Does the crypto repository have any kind of CI? It seems this could have been discovered automatically by CI.

[bradfitz]

@unclejack, it does. See https://build.golang.org/ at the bottom.

But that doesn't mean that the ssh package has perfect test coverage.

[CameronGo]

Hmm - is this really merged to master so that a go get golang.org/x/crypto/ssh would retrieve this fix? I ask b/c I re-ran my build scripts yesterday and my nightly batch processes still failed with this error?

[databus23]

@unclejack The supposed fix golang/crypto@77014cf is in master for 2 days and seems to be working for a lot of people (including me). Are you sure your tests used the latest master?

[hanwen]

@bradfitz - see

https://go.googlesource.com/crypto/+/ed03dad8de553b023b198606d52b6d968ad57b39

it would be great if these tests could run a couple of select platforms, such as linux.

@CameronGo did you supply -u to "go get" ?

[hanwen]

@unclejack - there are extensive unittests. Unfortunately, this issue is very sensitive to timing and scheduling. If you know of a way to exercise all the different orderings in which events can happen, let us know.

[unclejack]

@bradfitz @hanwen I'll try to come up with a test for this issue. The ssh package is critical in the context of a project. That's the reason why I've asked that question above. I didn't get a chance to try it out after the fix was merged.

I'll send a CL once that test is ready.

[CameronGo]

@hanwen - my jenkins scripts use the docker golang images, so the GOHOME is always empty when it launches, so the packages always get pulled. Still, I triggered another rebuild yesterday and my batch processes last night did not encounter any errors, so I'll keep an eye on it for the next couple of days, but I'm in good shape right now.

[yobert]

If this helps anyone trying to write a test case--- I didn't get this error until I started using an encrypted key via SSH agent. Loading the same key (but unencrypted) directly from disk worked just fine.

[hanwen]

This issue is fixed, so closing this bug.

[simon-s9]

When will this get to stable release? I'm on latest 1.8 (from golang.org) and the issue is still here.

[bradfitz]

@simon-s9, the "ssh" package is not part of the Go standard library, so whether you're on Go 1.8 or Go 1.6 is irrelevant. You just need to "go get -u" update the package.