Skip to content

SSH troubleshooting

Jump to bottom
Mark Scheel edited this page Nov 11, 2024 · 28 revisions

Symptoms

The symptom of most ssh problems is that you are asked for a password that doesn't exist. For example:

[scheel@frontera]$ git clone [email protected]:SimAnnex
[email protected]'s password:

Unfortunately there are many possible causes of such a problem. So the main strategy is to test each step of the ssh connection process individually (using some of the tips below) and eventually find out which step doesn't work and then fix it.

Troubleshooting ssh problems with ssh-add -l

The first thing I try is typing ssh-add -l (where the last character is a lowercase L, not the number 1). The output should be something like

3022 SHA256:rsDf7qJJQloXxItARsAst3Q5wVT5xzphwtezy7a7rQR scheel@shrike (RSA)

which identifies a key that your ssh agent holds. If you get output similar to the above, then your ssh-agent is fine. If instead it says something along the lines ofThe agent has no identities then something is wrong. It is usually one of two problems:

  1. If ssh-add -l fails on a machine you are physically touching, then the problem is that you have not enabled your key by typing in a passphrase since the last time you logged in. To fix this, type ssh-add and it will prompt you for a passphrase and enable your key. (If this doesn't work, you may need to first type eval `ssh-agent` ).

  2. If ssh-add -l fails on a remote machine (i.e. one that you are not physically touching), then either you have not enabled your key on your local machine (i.e. case 1. above, so check ssh-add -l on your local machine), or you don't have ssh-agent forwarding working correctly. The usual fix for ssh-agent forwarding problems is to add ForwardAgent yes to your $HOME/.ssh/config on your local machine. For example:

    Host frontera
  ForwardAgent yes

    Instead of enabling ssh forwarding for each machine individually, you can also enable it for all machines by adding the following to the end of your $HOME/.ssh/config:

    Host *
   ForwardAgent yes

Permissions

Running ls -ld $HOME/.ssh should reveal permissions drwx------. Similarly, running ls -l $HOME/.ssh should reveal that all the private files (your private keys, authorized_keys, known_hosts) should have permissions -rw-------. If you see different permissions than these, then ssh will probably fail, because it is a security risk (because, for example, everyone might be able read your private key which makes it not very private). Your public keys and config file will probably have permissions -rw-r--r--.

More things to try

If ssh-add -l gives reasonable output but the remote machine still asks for a password, the next thing I try is ssh -vv <remote_machine> on my local machine. The -vv option gives a lot of output that can usually tell you what is wrong.

no mutual signature algorithm

One common scenario: new versions of OSs on your local machine may deprecate using rsa for ssh-keygen by default. In this case, the remote machine (if it has an old key) will still ask for a password. If you type ssh -vv <remote_machine> on the local machine (or ssh -vv git@<remote_machine> if <remote_machine> happens to be sxs-archive or meistri where one usually connects as the git user), the error message you get would include a phrase send_pubkey_test: no mutual signature algorithm. One solution is to add -o PubkeyAcceptedKeyTypes=ssh-rsa to the ssh command, or add

PubkeyAcceptedKeyTypes=+ssh-rsa

to your local ~/.ssh/config in the section for the remote machine you are accessing. If you are trying to connect to a third machine (e.g. you are sitting at machine1, logging in to machine2, and then from machine2 you are trying to ssh into machine3), then you might need this line in the .ssh/config on machine2 as well as machine1. Alternatively, create a new SSH key using the newer and more secure ed25519 method unless you are trying to access CCEAnnex, SimulationAnnex, or the References repo, which doesn't recognize ed25519 keys. GitHub provides instructions.

no matching host key type found

Similar to the above problem, sometimes you get an error no matching host key type found. To fix this, add

HostKeyAlgorithms=+ssh-rsa

to your .ssh/config (again, if you are trying to connect from machine1->machine2->machine3 you might need this line in the .ssh/config of machine2 as well as machine1). Note that if you need the HostKeyAlgorithms line, you will most likely also need the PubkeyAcceptedKeyTypes line.

error in libcryptyo

This annoying error occurs because new versions of openssl disable sha1-signatures by default and provides no easy way to override. The fix is setting an environment variable pointing to Mark's config file. This depends on the machine.

On CaltechHPC:

export OPENSSL_CONF=/home/mascheel/software/openssl.conf

On mbot:

export OPENSSL_CONF=/home/fs01/spec1163/software/openssl.conf

If you are trying to set up an openssl.conf file on another cluster, the contents of the file should be:

.include /etc/ssl/openssl.cnf
[openssl_init]
alg_section = evp_properties
[evp_properties]
rh-allow-sha1-signatures = yes
Clone this wiki locally