[Remote-SSH Bug]: SSH Forward Agent not working anymore?

[pavel-francirek]

Is there an existing issue for this bug?

• I have searched the existing issues

Required Troubleshooting Steps

• I have followed these troubleshooting steps
• I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

No response

Expected Behavior

SSH operation on remote stopped working in v0.110.1 update:

$ git pull
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

Works in v0.110.0 (after downgrade) and older:

$ git pull
Already up to date.

VSCode runs on Windows 10, remote is Linux Ubuntu (different LTS versions).

Steps To Reproduce

No response

Remote-SSH Log

Remote-SSH Log

[09:23:17.216] Log Level: 2
[09:23:17.222] VS Code version: 1.88.0
[09:23:17.222] Remote-SSH version: [email protected]
[09:23:17.223] win32 x64
[09:23:17.228] SSH Resolver called for "ssh-remote+example", attempt 1
[09:23:17.229] "remote.SSH.useLocalServer": false
[09:23:17.229] "remote.SSH.useExecServer": true
[09:23:17.229] "remote.SSH.showLoginTerminal": false
[09:23:17.229] "remote.SSH.remotePlatform": {"example":"linux"}
[09:23:17.229] "remote.SSH.path": C:\Windows\System32\OpenSSH\ssh.exe
[09:23:17.229] "remote.SSH.configFile": undefined
[09:23:17.230] "remote.SSH.useFlock": true
[09:23:17.231] "remote.SSH.lockfilesInTmp": false
[09:23:17.231] "remote.SSH.localServerDownload": auto
[09:23:17.231] "remote.SSH.remoteServerListenOnSocket": false
[09:23:17.231] "remote.SSH.showLoginTerminal": false
[09:23:17.231] "remote.SSH.defaultExtensions": []
[09:23:17.231] "remote.SSH.loglevel": 2
[09:23:17.232] "remote.SSH.enableDynamicForwarding": true
[09:23:17.232] "remote.SSH.enableRemoteCommand": true
[09:23:17.232] "remote.SSH.serverPickPortsFromRange": {}
[09:23:17.232] "remote.SSH.serverInstallPath": {}
[09:23:17.238] SSH Resolver called for host: example
[09:23:17.238] Setting up SSH remote "example"
[09:23:17.242] Using commit id "5c3e652f63e798a5ac2f31ffd0d863669328dc4c" and quality "stable" for server
[09:23:17.245] Install and start server if needed
[09:23:17.254] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V"
[09:23:17.288] > OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

[09:23:17.292] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 63708 example bash
[09:23:17.294] Terminal shell path: C:\Windows\System32\cmd.exe
[09:23:18.521] > �]0;C:\Windows\System32\cmd.exe�
[09:23:18.521] Got some output, clearing connection timeout
[09:23:20.686] > ac26998f326f: running```

</p>
</details>


### Anything else?

_No response_

[pavel-francirek]

NOTE: Disabling remote.SSH.useExecServer didn't work.
NOTE2: Connection uses ProxyJump. Definition is in user's ssh/config and works with Windows native ssh client without issue.

[danielhass]

@pavel-francirek - I ran into a similar situation like you did. I want to check if we are really hitting the same bug or a slight variation as for me after disabling remote.SSH.useExecServer it seems to work fine again.

Cloud you check with v0.110.1 if the SSH_AUTH_SOCK on the remote system after establishing the remote connection is pointing to an existing socket file?

For me this want the case and this variable was basically stale after a the second remote connection to a freshly installed remove server. With stale meaning that it worked the first time pointing to a existing forwarded agent socket file and afterwards stayed on this exact value and didn't update on subsequent logins/remote connections. Hence this was causing e.g. git pull errors for me.

However it is important to mention that the actual agent forwarding was working fine in all cases. But the shell environment wasn't updated properly to make use of the forwarded socket file.

You should be able to check the situation on the remote system with the following commands:

# list the current SSH_AUTH_SOCK variables
$ env | grep -i SSH_AUTH

# list the currently forwarded socket files
$ ls /tmp | grep -i ssh

[pavel-francirek]

@danielhass
You are right. There is non-existing path in SSH_AUTH_SOCK variable. Both with or without remote.SSH.useExecServer.
Example:

$ env | grep -i SSH_AUTH
SSH_AUTH_SOCK=/run/user/1000/vscode-ssh-auth-sock-694344400
$ ls -l /run/user/1000/vscode-ssh-auth-sock-694344400
lrwxrwxrwx 1 user user 32 Apr 11 16:14 /run/user/1000/vscode-ssh-auth-sock-694344400 -> /tmp/ssh-gZKP5lVtQ5/agent.606759
$ ls -l /tmp/ssh-gZKP5lVtQ5/agent.606759
ls: cannot access '/tmp/ssh-gZKP5lVtQ5/agent.606759': No such file or directory

[danielhass]

@pavel-francirek could you try the following to see if this resolves the issue for you? (helped on my side)

• close your VSCode instances
• remove the server instance on the remote system (the default should be ~/.vscode-server, but maybe double check that)
• disable remote.SSH.useExecServer
• connect to the remote machine again (now the server should be freshly pulled and installed on the remote side)

Afterwards, at least for me, this issue has been gone. For me it looks like as there is something persisted on the remote side that is causing this issue. And once that's there, a change of the exec option doesn't help.

[jchen42703]

Had the same issue. On Ubuntu 22.0.4, setting "remote.SSH.useExecServer": false, in my local vscode config made it work for me.

[pavel-francirek]

@danielhass

@pavel-francirek could you try the following to see if this resolves the issue for you? (helped on my side)
* close your VSCode instances
* remove the server instance on the remote system (the default should be ~/.vscode-server, but maybe double check that)

Thank you for good hint. "Remote-SSH: Uninstall VS Code Server from Host..." from Command Palette was the right solution. Had to do it on all dev servers and had to set remote.SSH.useExecServer: true or it didn't work with Windows ssh-agent but everything works again now.

[hellt]

Hi all,
I was having the same issue with the stale SSH_AUTH_SOCK env var since a couple of months, and while killing the remove vscode server fixes that, I doubt that this workaround warrants closing this issue?

Shouldn't this be promoted to a software issue and tracked?

[danielhass]

@hellt I would also vote for letting this issue open and at least for a maintainer to look at this situation.

@pavel-francirek would you mind re-open it?

[hellt]

And I would appreciate if anyone knows what are the implications of setting remote.ssh use exec server to false?

The description of this setting is missing in the recent versions.

[mgenereu]

@hellt , here's some docs: https://github.com/microsoft/vscode-remote-release/wiki/Remote-SSH-troubleshooting#remotesshuseexecserver---exec-server-mode
Not a fix but some insight?

[dobos]

I'm seeing the same issue when I work remotely from home. I guess the difference is that my client often reconnects when the machine goes to sleep. My office PC is set up not to sleep ever.

[SiestaCat]

@pavel-francirek could you try the following to see if this resolves the issue for you? (helped on my side)

• close your VSCode instances
• remove the server instance on the remote system (the default should be ~/.vscode-server, but maybe double check that)
• disable remote.SSH.useExecServer
• connect to the remote machine again (now the server should be freshly pulled and installed on the remote side)

Afterwards, at least for me, this issue has been gone. For me it looks like as there is something persisted on the remote side that is causing this issue. And once that's there, a change of the exec option doesn't help.

@pavel-francirek could you try the following to see if this resolves the issue for you? (helped on my side)

• close your VSCode instances
• remove the server instance on the remote system (the default should be ~/.vscode-server, but maybe double check that)
• disable remote.SSH.useExecServer
• connect to the remote machine again (now the server should be freshly pulled and installed on the remote side)

Afterwards, at least for me, this issue has been gone. For me it looks like as there is something persisted on the remote side that is causing this issue. And once that's there, a change of the exec option doesn't help.

Hello. I'm not using Remote SSH extension, i'm using Dev Containers. How to solve this issue? Thanks