[Remote-SSH Bug]: SSH agent forwarding broken after reconnect

[pelmers-db]

• VSCode Version: 1.93.1 (Universal)
• Local OS Version: 14.6.1 (23G93)
• Remote OS Version: Ubuntu 18
• Remote Extension/Connection Type: Server

this is basically a duplicate submission of #9662

That issue was closed as completed, however I have observed the same behavior in latest release of remote ssh extension. [10:53:17.474] Remote-SSH version: [email protected]

Steps to Reproduce:

1. work normally for a while
2. go away for a while
3. re-open laptop, VS Code reconnects and most things work
4. cannot git pull/git push, since those use agent keys

Does this issue occur when you try this locally?: no

[pelmers-db]

like the previous issue, this one goes away when setting remote.SSH.useExecServer: false

[pelmers-db]

Repro'd again. On a MacOS client it's quite easy to do:

1. put the laptop to sleep
2. return, open the terminal

$ ssh-add -l                      
Error connecting to agent: No such file or directory

3. the same happens even if I open a new terminal, or close all terminals and then open more

Further investigation suggests it's something in the extension that's injecting old environment variables into the shell session.

Evidence:

1. observe output of remote - ssh extension from when I returned:

[12:27:55.639] > Found existing installation at /home/peter.elmers/.vscode-server...
[12:27:55.640] > Starting VS Code CLI...
> printenv:
>     USER=peter.elmers
>     SSH_CLIENT=130.41.208.2 47952 22
>     MOTD_SHOWN=pam
>     HOME=/home/peter.elmers
>     OLDPWD=/home/peter.elmers
>     DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
>     LOGNAME=peter.elmers
>     _=/usr/bin/sh
>     PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
>     LANG=C.UTF-8
>     SSH_AUTH_SOCK=/tmp/ssh-wnvVKJVza5/agent.73938
>     SHELL=/usr/bin/zsh
>     PWD=/home/peter.elmers
>     SSH_CONNECTION=130.41.208.2 47952 10.91.21.232 22
[12:27:55.640] > Removing old logfile at /home/peter.elmers/.vscode-server/.cli.38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40.log
[12:27:55.643] > Spawned remote CLI: 73963

2. in a broken terminal:

$ env | grep SSH                  
SSH_CLIENT=130.41.208.2 54400 22
SSH_AUTH_SOCK=/tmp/ssh-fJiZOJuhV7/agent.9790
SSH_CONNECTION=130.41.208.2 54400 10.91.21.232 22

3. observe! the printed SSH_AUTH_SOCK and the remote - ssh new SSH_AUTH_SOCK do not match!
4. manually inject the new SSH_AUTH_SOCK, and it prints my keys

$ SSH_AUTH_SOCK=/tmp/ssh-wnvVKJVza5/agent.73938 ssh-add -l
4096 SHA256:blahblah peter.elmers@mylaptophostname (RSA)

[pelmers-db]

update: this can also happen when remote.SSH.useExecServer: false, which I thought fixed the problem. I believe the issue is a buggy interaction with another vs code setting, terminal.integrated.enablePersistentSessions. When that is set to true, then vs code will restore terminal state on reconnect. Apparently the state also includes incorrect SSH_AUTH_SOCK environment.

In this case the fix is to open a new terminal.

tl;dr:
remote.SSH.useExecServer: true: completely broken
remote.SSH.useExecServer: false: the restored terminal is broken, new terminals are fixed

[ehoogeveen-medweb]

FWIW, we also experience the exact same issue on Windows 11 (connecting to a Linux remote) - we have to disable remote.SSH.useExecServer to make it work at all, and still sometimes need to close the terminal with the garbage can icon and open a new one to restore functionality.

So this is not an issue specific to MacOS - perhaps it's less common on Windows because the SSH agent is disabled by default, so agent forwarding takes more steps to set up. Perhaps there are other specific requirements to make the bug happen, but I haven't found any.

[danielhass]

I stumbled into this behavior too, currently using v0.115.0 on Windows 11 with a Linux Remote.

The problem is reproducible in a consistent manner and vanishes when switching to remote.SSH.useExecServer: false with reinstall of the remote server.

As I was already involved in #9819 this looks like a similar case at least from the users perspective.

[danielhass]

Just an update from my side. I'm still able to reproduce this issue with the latest pre-release version of the Remote SSH extension at the time of writing this comment v0.116.2024111219 when using the remote.SSH.useExecServer: true setting.

[joshspicer]

This should be fixed in the latest pre-release (released earlier today 0.116.2024112219).

With exec server enabled you should see SSH_AUTH_SOCK set to a socket with a name like vscode-ssh-auth-sock-xxxxxx

josh@devbox:~$ ssh-add -l
256 SHA256:X9xxXXXXxxXxx9xx9xx9xx9xxXxXXXXx999x9XxxxxxX [email protected] (ED25519)
josh@devbox:~$ echo $SSH_AUTH_SOCK 
/run/user/1000/vscode-ssh-auth-sock-231840959
josh@devbox:~$ 

Please let me know if you're seeing otherwise and include your logs (+ repro steps if relevant). Thanks for folks' patience!

[ehoogeveen-medweb]

I switched to pre-release and am still having issues, but the extension version is 0.116.2024111219. 1112 instead of 1122, so I'm guessing it's an older version? There's no option to update.

Edit: Trying it in VSCode Insiders now (thanks Daniel, I hadn't considered it) and it does appear to be working!

[danielhass]

@joshspicer - from a first look I can confirm that 0.116.2024112219 seems to fix the issue. However more extensive daily driving this version is required from my side.

As @ehoogeveen-medweb pointed out with the current stable VSCode build it is only possible to bump up the pre-release to 0.116.2024111219. I tried to install the extension via .vsix and ran into this which probably explains the issue:

Afterwards I tried 0.116.2024112219 via the VSCode Insider build and I wasn't able to reproduce the issue there.

[joshspicer]

Apologies for not mentioning, you'll need to test in insiders since this pre-release other changes were made that require the newest VS Code APIs.

Thanks very much for the initial testing @ehoogeveen-medweb and @danielhass! After some more extended use, please let me know if you encounter any further issues

[alexr00]

What's the best way to verify this?

[joshspicer]

This has been validated by two community members, so i'm confident it has helped. Here are the steps:
 

Repro steps

• Use VS Code Insiders and the latest Remote - SSH pre-release
• Use the default Remote - SSH settings (which is useExecServer == true)
• Ensure locally with ssh-add -l that you have an some key and and ssh agent running

[/tmp]$ ssh-add -l
256 SHA256:U3qgLJIW53kafmc1ksbAdk7777j87w6LjghiA mykey (ED25519)

• You can add your key with ssh-add <key-path>
• Connect to a host, being sure that ForwardAgent is enabled in that host's configuration

  Host myHost
    HostName 1.2.3.4
    User josh
    ForwardAgent yes

• Check that the symlink is as expected
• Confirm that ssh-add -l on the remote has your key
• Confirm that after some time, a reconnect, etc that things don't become broken (that symlink points to the wrong spot)

[rzhao271]

Marking as verified based on the comments above.