Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH Login Issues from the shell via PEM cert, but not with Ansible playbook or inspec shell #703

Closed
aaronlippold opened this issue Sep 12, 2024 · 13 comments
Assignees

Comments

@aaronlippold
Copy link

Hello,

I am having an odd issue when I run the https://github.com/ansible-lockdown/RHEL9-STIG on the SPEL image in AWS that SSH login with a PEM cert stops working in the shell.

Oddly, if I run the playbook using the same cert and user, ansible is able to connect and run the playbook.

Has anyone run into an issue like this before?

➜  redhat-enterprise-linux-9-stig-baseline git:(main*)ssh -i .kitchen/hardened-rhel-9.pem -F /dev/null -vvvvv -p 22 \
    -o UserKnownHostsFile=/dev/null \
    -o Compression=no \        
    -o ServerAliveInterval=60 \
    -o ConnectTimeout=15 \                      
    -o PreferredAuthentications=none,publickey \
    -o StrictHostKeyChecking=no \
    [email protected]

OpenSSH_9.7p1, LibreSSL 3.3.6
debug1: Reading configuration data /dev/null
debug2: resolve_canonicalize: hostname 54.242.90.186 is address
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug3: channel_clear_timeouts: clearing
debug3: ssh_connect_direct: entering
debug1: Connecting to 54.242.90.186 [54.242.90.186] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 15000 ms remain after connect
debug1: identity file .kitchen/hardened-rhel-9.pem type -1
debug1: identity file .kitchen/hardened-rhel-9.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
debug1: compat_banner: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 54.242.90.186:22 as 'ec2-user'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: no algorithms matched; accept original
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: [email protected],curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c,[email protected]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,[email protected]
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256
debug2: ciphers ctos: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: ciphers stoc: [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
debug2: MACs ctos: [email protected],hmac-sha2-256,[email protected],hmac-sha2-512,[email protected],[email protected],hmac-sha1,[email protected]
debug2: MACs stoc: [email protected],hmac-sha2-256,[email protected],hmac-sha2-512,[email protected],[email protected],hmac-sha1,[email protected]
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug3: kex_choose_conf: will use strict KEX ordering
debug1: kex: algorithm: ecdh-sha2-nistp256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 54.242.90.186 port 22
@lorengordon
Copy link
Member

Hi @aaronlippold, I'm not entirely sure exactly what you're running into, as we do not use the ansible-lockdown scripts. But the SPEL images do have FIPS enabled out of the box, and that does impose a number of restrictions on the allowed ciphers. It might be that your ssh cert is not using an allowed cipher. For example, ED25519 keys are not allowed when FIPS is enabled. Also, SHA1 ciphers will be rejected. As well as keys with a length less than 2048 bits.

One thing from the logs:

debug1: Authenticating to 54.242.90.186:22 as 'ec2-user'

The default user for spel images is maintuser, not ec2-user. Perhaps you are using your own images and changing the default user, but if not, then try using maintuser.

@aaronlippold
Copy link
Author

aaronlippold commented Sep 14, 2024 via email

@aaronlippold
Copy link
Author

I did a little hacking and found that this command

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o LogLevel=VERBOSE -o [email protected],aes256-ctr,[email protected],aes128-ctr -o [email protected],hmac-sha2-256,[email protected],hmac-sha2-512 -o KexAlgorithms=ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512 -p 22 -i .kitchen/hardened-rhel-9.pem maintuser@<myip>

works

The SSHD runtime is, the thing that jumps out to me is the `gssapikexalgorithms'

I can't find anything in the ansible with respect to gssapikeyalgorithms but at least I can ssh into it now and not have to try to debug over the inspec shell

Any thing jump out to you guys on this?

port 22
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam yes
logingracetime 120
x11displayoffset 10
x11maxdisplays 1000
maxauthtries 6
maxsessions 10
clientaliveinterval 600
clientalivecountmax 1
requiredrsasize 2048
streamlocalbindmask 0177
permitrootlogin no
ignorerhosts yes
ignoreuserknownhosts yes
hostbasedauthentication no
hostbasedusesnamefrompacketonly no
pubkeyauthentication yes
kerberosauthentication no
kerberosorlocalpasswd yes
kerberosticketcleanup yes
kerberosuniqueccache no
kerberosusekuserok yes
gssapienablek5users no
gssapiauthentication no
gssapicleanupcredentials no
gssapikeyexchange no
gssapistrictacceptorcheck yes
gssapistorecredentialsonrekey no
gssapikexalgorithms gss-group14-sha256-,gss-group16-sha512-,gss-nistp256-sha256-,gss-curve25519-sha256-,gss-
group14-sha1-,gss-gex-sha1-
passwordauthentication yes
kbdinteractiveauthentication no
printmotd no
printlastlog yes
x11forwarding yes
x11uselocalhost yes
permittty yes
permituserrc yes
strictmodes yes
tcpkeepalive yes
permitemptypasswords no
compression no
gatewayports no
usedns no
allowtcpforwarding yes
allowagentforwarding yes
disableforwarding no
allowstreamlocalforwarding yes
streamlocalbindunlink no
fingerprinthash SHA256
exposeauthinfo no
pidfile /var/run/sshd.pid
modulifile /etc/ssh/moduli
xauthlocation /usr/bin/xauth
ciphers [email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
macs [email protected],hmac-sha2-256,[email protected],hmac-sha2-512,hmac-sha1-etm@o
penssh.com,[email protected],hmac-sha1,[email protected]
banner /etc/issue
forcecommand none
chrootdirectory none
trustedusercakeys none
revokedkeys none
securitykeyprovider internal
authorizedprincipalsfile none
versionaddendum none
authorizedkeyscommand none
authorizedkeyscommanduser none
authorizedprincipalscommand none
authorizedprincipalscommanduser none
hostkeyagent none
kexalgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,
diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
casignaturealgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512
hostbasedacceptedalgorithms [email protected],[email protected]
m,[email protected],[email protected],[email protected]
om,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256
hostkeyalgorithms ecdsa-sha2-nistp256,[email protected],ecdsa-sha2-nistp384,ecdsa-sha
[email protected],ecdsa-sha2-nistp521,[email protected],rsa-sha2-256,rs
[email protected],rsa-sha2-512,[email protected]
pubkeyacceptedalgorithms ecdsa-sha2-nistp256,[email protected],ecdsa-sha2-nistp384,ec
[email protected],ecdsa-sha2-nistp521,[email protected],rsa-sha2
-256,[email protected],rsa-sha2-512,[email protected]
loglevel VERBOSE
banner /etc/issue
forcecommand none
chrootdirectory none
trustedusercakeys none
revokedkeys none
securitykeyprovider internal
authorizedprincipalsfile none
versionaddendum none
authorizedkeyscommand none
authorizedkeyscommanduser none
authorizedprincipalscommand none
authorizedprincipalscommanduser none
hostkeyagent none
kexalgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,
diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
casignaturealgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512
hostbasedacceptedalgorithms [email protected],[email protected]
m,[email protected],[email protected],[email protected]
om,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256
hostkeyalgorithms ecdsa-sha2-nistp256,[email protected],ecdsa-sha2-nistp384,ecdsa-sha
[email protected],ecdsa-sha2-nistp521,[email protected],rsa-sha2-256,rs
[email protected],rsa-sha2-512,[email protected]
pubkeyacceptedalgorithms ecdsa-sha2-nistp256,[email protected],ecdsa-sha2-nistp384,ec
[email protected],ecdsa-sha2-nistp521,[email protected],rsa-sha2
-256,[email protected],rsa-sha2-512,[email protected]
loglevel VERBOSE
syslogfacility AUTHPRIV
authorizedkeysfile .ssh/authorized_keys
hostkey /etc/ssh/ssh_host_rsa_key
hostkey /etc/ssh/ssh_host_ecdsa_key
hostkey /etc/ssh/ssh_host_ed25519_key
authenticationmethods any
subsystem sftp /usr/libexec/openssh/sftp-server
maxstartups 10:30:100
persourcemaxstartups none
persourcenetblocksize 32:128
permittunnel no
ipqos af21 cs1
rekeylimit 1073741824 3600
permitopen any
permitlisten any
permituserenvironment no
pubkeyauthoptions none

@lorengordon
Copy link
Member

@ferricoxide if you get a minute, could you offer any suggestions?

@aaronlippold
Copy link
Author

Added Note:

I also added this to my ~/.ssh/config - which is a combination of the both the defaults on my ssh client on osx and the required STIG required ciphers, MACs, and key exchange algorithms. However this feels more like a 'workaround' than a solution.

Host *
  Ciphers 3des-cbc,aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],[email protected],[email protected],aes256-ctr,[email protected],aes128-ctr
  MACs hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512,hmac-md5,hmac-md5-96,[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,[email protected],hmac-sha2-512
  KexAlgorithms diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,[email protected],[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512

This may be an effect of enabling FIPS, but updating the client side ssh config has not been something I have have to do before and I have been connecting to DOD boxing a few many years :)

@ferricoxide
Copy link
Member

Having to configure the ssh-client configuration is something that seems to come along every couple RHEL generations. I'd assumed that some of it was due the march of ever-tightening security controls (e.g., EL8 no longer accepts anything less than a SHA256 key if you're using RSAv2 and wholly disables support for most of the other encryption-families that EL 6 & 7 supported).

At any rate, last time I had to configure Ciphers/MACs in my SSH client was RHEL 5 time-frame (2010ish). Pretty sure the reason I had to do it, at all, was that I was using generic SSH clients. Since switching away from using either the local OS's native SSH client, Cygwin/X's client or PuTTY's, it's basically been a non-problem.

That said, avoidance of those came due a change in my personal workflow: I no longer connect directly from my local OS. Instead, I run ELx VMs on my local host and connect from those VMs. I would assume that, since I use "latest and greatest" for my local VMs, changes caused by posture-deltas between the remote host and the local clients are generally eliminated (or, where deltas exist, the SSH client I'm using is generally going to be more fascist than the remote server's configuration).

@aaronlippold
Copy link
Author

aaronlippold commented Sep 16, 2024 via email

@lorengordon
Copy link
Member

I feel like OSX is infamous for including out-of-date clients and shells. Really need to manage the dev environment and take full control, not rely on OSX as much.

@ferricoxide
Copy link
Member

Honestly, you're the first person reporting this. That said I'd imagine that's influenced by two things:

  1. Most users of spel AMIs are using the EL 8 AMIs, not the EL 9 AMIs (we only just recently started making EL9 hardening-content available and have no spel-using organizations that have made us aware that they're even beginning to think about life beyond EL 8).
  2. Most users of spel AMIs aren't doing so via direct connection from OSX (most of the organizations we have explicit knowledge of are primarily Windows-based organizations)

Neither of which is to be confused with me saying there's nothing to be put in the documentation, just that, absent indications to the contrary, yours appears to be a very corner-case scenario. I know that in the past, we've had issues with OSX-users' execution of spel and related projects' BASH-based logic as the default version of BASH in OSX has been considerably behind "current" – even moreso than that in Red Hat's major-releases. I would be curious to see what your results were if using a more up-to-date SSH client (i.e., your mention of using brew).

@aaronlippold
Copy link
Author

aaronlippold commented Sep 16, 2024 via email

@aaronlippold
Copy link
Author

aaronlippold commented Sep 16, 2024 via email

@ferricoxide
Copy link
Member

@aaronlippold:

Since this issue was a functionality issue that seems to be something more-appropriately addressed as a documentation issue, I opened a separate (documentation) issue (#705) to track it. I'm going to close this issue in favor of the documentation-issue.

@aaronlippold
Copy link
Author

Sounds great

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants