Could not load source ...: Incorrect format of 'source' message..

[BMurri]

Issue Description

This was first reported here. I did change the paths for privacy, but I may as well unmask that at this point.

Steps to Reproduce

Implement c# console app (net5). Create task publishAll that builds project, copies bin/Debug/net5.0 folder to ~/cromwell/bin and workspace folder (except for obj, bin, and .git) to ~/cromwell/code. Add following to launch.json under configurations:

    {
        "name": ".NET Core Launch (remote console)",
        "type": "coreclr",
        "request": "launch",
        "preLaunchTask": "publishAll",
        "program": "~/cromwell/bin/deploy-cromwell-on-azure.dll",
        "args": [],
        "cwd": "~/cromwell/bin",
        "stopAtEntry": false,
        "console": "integratedTerminal",
        "pipeTransport": {
            "pipeCwd": "${workspaceFolder}",
            "pipeProgram": "ssh.exe",
            "pipeArgs": ["-T" ,"EntryInSSHConfig"[], "${debuggerCommand}"],
            "debuggerPath": "~/vsdbg/vsdbg"
        },
        "sourceFileMap": {
            "${workspaceFolder}": "~/cromwell/code"
        },
        "logging": {
            "engineLogging": true
        }

Set breakpoint at line 96 in Deployer.cs file, select .NET Core Launch (remote console) configuration, hit green triangle.

Expected Behavior

debugger stops at line 96 in Deployer.cs file, vscode shows Deployer.cs file around that line, breakpoint shows red in column to left of line numbers where it was red before launching.

Actual Behavior

• new tab opens in editor, showing \remote\sources\path\Deployer.cs in bold+italics, with content: Could not load source \remote\sources\path\Deployer.cs: Incorrect format of 'source' message..
• Original tab in editor does not show breakpoint as active.
• Breakpoints list shows that there is at least one active breakpoint in \remote\sources\path\Deployer.cs

Note that vscode is running in Windows, and debugger is running in Linux.

Note also that the remote extensions do me no good because I need my code and my debugger at different locations (the source copy to remote is a concession to the fact that the debugger simply would refuse to honor breakpoints when it can't read the source file "locally").

If it helps. I had captured the following engine logging (trimmed to what looked most pertinent):

-> (C) {"command":"setBreakpoints","arguments":{"source":{"name":"Deployer.cs","path":"${workspaceFolder}\Deployer.cs","checksums":[{"algorithm":"SHA1","checksum":"6df7c4f3a3bdd7b19f49ae99fb427f6984c67668"},{"algorithm":"SHA256","checksum":"a8c9a3ddb3949779c5e3b103abeee4b4f0b39223311b9b7ce9b1259b8a01dc69"},{"algorithm":"SHA1","checksum":"5f4ca8a154a8be70898341534d8e2d66653cac0c"},{"algorithm":"SHA256","checksum":"676841396975f2bcfa5d96ded90ced39685ab2aaac302427535b3d0a7d3ef139"}]},"lines":[96],"breakpoints":[{"line":96}],"sourceModified":false},"type":"request","seq":3}
<- (R) {"seq":6,"type":"response","request_seq":3,"success":true,"command":"setBreakpoints","body":{"breakpoints":[{"id":1,"verified":false,"message":"The breakpoint is pending and will be resolved when debugging starts.","line":96}]}}
...
<- (E) {"seq":20,"type":"event","event":"breakpoint","body":{"reason":"changed","breakpoint":{"id":1,"verified":true,"source":{"name":"Deployer.cs","path":"\\remote\\sources\\path\\Deployer.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"a8c9a3ddb3949779c5e3b103abeee4b4f0b39223311b9b7ce9b1259b8a01dc69"}]},"line":96,"endLine":96,"offset":0}}}
...
<- (E) {"seq":92,"type":"event","event":"breakpoint","body":{"reason":"changed","breakpoint":{"id":1,"verified":true,"source":{"name":"Deployer.cs","path":"\\remote\\sources\\path\\Deployer.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"a8c9a3ddb3949779c5e3b103abeee4b4f0b39223311b9b7ce9b1259b8a01dc69"}]},"line":96,"endLine":96,"instructionReference":"0x00007F2BBBDE0E7D","offset":0}}}
<- (E) {"seq":93,"type":"event","event":"stopped","body":{"reason":"breakpoint","threadId":5332,"allThreadsStopped":true,"hitBreakpointIds":[1],"source":{"name":"Deployer.cs","path":"\\remote\\sources\\path\\Deployer.cs\\Deployer.cs"},"line":96,"column":13}}-> (C) {"command":"threads","type":"request","seq":9}
<- (R) {"seq":94,"type":"response","request_seq":9,"success":true,"command":"threads","body":{"threads":[{"id":5332,"name":"<No Name>"}]}}
-> (C) {"command":"stackTrace","arguments":{"threadId":5332,"startFrame":0,"levels":20},"type":"request","seq":10}
-> (C) {"command":"stackTrace","arguments":{"threadId":5332,"startFrame":0,"levels":20},"type":"request","seq":11}
<- (R) {"seq":95,"type":"response","request_seq":10,"success":true,"command":"stackTrace","body":{"stackFrames":[{"id":1000,"name":"executable.dll!CromwellProjectNamespace.Deployer.DeployAsync() Line 96","source":{"name":"Deployer.cs","path":"\\remote\\sources\\path\\Deployer.cs\\Deployer.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"a8c9a3ddb3949779c5e3b103abeee4b4f0b39223311b9b7ce9b1259b8a01dc69"}]},"line":96,"column":13,"endLine":96,"endColumn":50,"instructionPointerReference":"0x00007F2BBBDE0E7D","moduleId":1001},{"id":1001,"name":"executable.dll!CromwellProjectNamespace.Program.InitializeAndDeployAsync(string[] args) Line 37","source":{"name":"Program.cs","path":"\\remote\\sources\\path\\Program.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"6a3707980792e6383361196e1c0bcad30b5e67456480080addf7d8d099c69ef3"}]},"line":37,"column":13,"endLine":37,"endColumn":79,"instructionPointerReference":"0x00007F2BBBDC7EA6","moduleId":1001},{"id":1002,"name":"executable.dll!CromwellProjectNamespace.Program.Main(string[] args) Line 13","source":{"name":"Program.cs","path":"\\remote\\sources\\path\\Program.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"6a3707980792e6383361196e1c0bcad30b5e67456480080addf7d8d099c69ef3"}]},"line":13,"column":13,"endLine":13,"endColumn":50,"instructionPointerReference":"0x00007F2BBBDC6BBB","moduleId":1001},{"id":1003,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"}],"totalFrames":4}}
<- (R) {"seq":96,"type":"response","request_seq":11,"success":true,"command":"stackTrace","body":{"stackFrames":[{"id":1000,"name":"executable.dll!CromwellProjectNamespace.Deployer.DeployAsync() Line 96","source":{"name":"Deployer.cs","path":"\\remote\\sources\\path\\Deployer.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"a8c9a3ddb3949779c5e3b103abeee4b4f0b39223311b9b7ce9b1259b8a01dc69"}]},"line":96,"column":13,"endLine":96,"endColumn":50,"instructionPointerReference":"0x00007F2BBBDE0E7D","moduleId":1001},{"id":1001,"name":"executable.dll!CromwellProjectNamespace.Program.InitializeAndDeployAsync(string[] args) Line 37","source":{"name":"Program.cs","path":"\\remote\\sources\\path\\Program.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"6a3707980792e6383361196e1c0bcad30b5e67456480080addf7d8d099c69ef3"}]},"line":37,"column":13,"endLine":37,"endColumn":79,"instructionPointerReference":"0x00007F2BBBDC7EA6","moduleId":1001},{"id":1002,"name":"executable.dll!CromwellProjectNamespace.Program.Main(string[] args) Line 13","source":{"name":"Program.cs","path":"\\remote\\sources\\path\\Program.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"6a3707980792e6383361196e1c0bcad30b5e67456480080addf7d8d099c69ef3"}]},"line":13,"column":13,"endLine":13,"endColumn":50,"instructionPointerReference":"0x00007F2BBBDC6BBB","moduleId":1001},{"id":1004,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"}],"totalFrames":4}}
-> (C) {"command":"source","arguments":{"sourceReference":0,"source":{"name":"Deployer.cs","path":"\\remote\\sources\\path\\Deployer.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"a8c9a3ddb3949779c5e3b103abeee4b4f0b39223311b9b7ce9b1259b8a01dc69"}]}},"type":"request","seq":12}
<- (E) {"seq":97,"type":"event","event":"output","body":{"category":"telemetry","output":"VS/Diagnostics/Debugger/vsdbg/CommandFailed","data":{"VS.Diagnostics.Debugger.vsdbg.ErrorCode":21000,"VS.Diagnostics.Debugger.vsdbg.Command":"source","VS.Diagnostics.Debugger.vsdbg.AdapterId":"coreclr","VS.Diagnostics.Debugger.vsdbg.Distribution.Version":"20.04","VS.Diagnostics.Debugger.vsdbg.Distribution.Name":"ubuntu","VS.Diagnostics.Debugger.vsdbg.OSFamily":"Linux","VS.Diagnostics.Debugger.vsdbg.Version":"17.0.10712.2 commit:70a83505117741ba30f92c713a4bb0d0395c3197"}}}
<- (R) {"seq":98,"type":"response","request_seq":12,"success":false,"command":"source","message":"Incorrect format of 'source' message."}

Logs

OmniSharp log

Starting OmniSharp server at 10/26/2021, 1:27:14 AM Target: c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure

OmniSharp server started.
Path: c:\Users\v-blairmurri.vscode\extensions\ms-dotnettools.csharp-1.23.16.omnisharp\1.37.16\OmniSharp.exe
PID: 3124

Starting OmniSharp on Windows 6.2.9200.0 (x64)
info: OmniSharp.Services.DotNetCliService
Checking the 'DOTNET_ROOT' environment variable to find a .NET SDK
info: OmniSharp.Services.DotNetCliService
Using the 'dotnet' on the PATH.
info: OmniSharp.Services.DotNetCliService
DotNetPath set to dotnet
info: OmniSharp.MSBuild.Discovery.MSBuildLocator
Located 2 MSBuild instance(s)
1: Visual Studio Community 2019 16.11.31729.503 16.11.1 - "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin"
2: StandAlone 17.0.0 - "c:\Users\v-blairmurri.vscode\extensions\ms-dotnettools.csharp-1.23.16.omnisharp\1.37.16.msbuild\Current\Bin"
info: OmniSharp.MSBuild.Discovery.MSBuildLocator
Registered MSBuild instance: Visual Studio Community 2019 16.11.31729.503 16.11.1 - "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin"
info: OmniSharp.WorkspaceInitializer
Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.CSharpFormattingWorkspaceOptionsProvider, Order: 0
info: OmniSharp.WorkspaceInitializer
Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.Completion.CompletionOptionsProvider, Order: 0
info: OmniSharp.WorkspaceInitializer
Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.RenameWorkspaceOptionsProvider, Order: 100
info: OmniSharp.WorkspaceInitializer
Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.ImplementTypeWorkspaceOptionsProvider, Order: 110
info: OmniSharp.WorkspaceInitializer
Invoking Workspace Options Provider: OmniSharp.Roslyn.CSharp.Services.BlockStructureWorkspaceOptionsProvider, Order: 140
info: OmniSharp.Cake.CakeProjectSystem
Detecting Cake files in 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure'.
info: OmniSharp.Cake.CakeProjectSystem
Could not find any Cake files
info: OmniSharp.MSBuild.ProjectSystem
No solution files found in 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure'
info: OmniSharp.MSBuild.ProjectManager
Queue project update for 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure\deploy-cromwell-on-azure.csproj'
info: OmniSharp.Script.ScriptProjectSystem
Detecting CSX files in 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure'.
info: OmniSharp.Script.ScriptProjectSystem
Could not find any CSX files
info: OmniSharp.WorkspaceInitializer
Configuration finished.
info: OmniSharp.Stdio.Host
Omnisharp server running using Stdio at location 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure' on host 18472.
info: OmniSharp.MSBuild.ProjectManager
Loading project: c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure\deploy-cromwell-on-azure.csproj
info: OmniSharp.MSBuild.ProjectManager
Successfully loaded project file 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure\deploy-cromwell-on-azure.csproj'.
info: OmniSharp.MSBuild.ProjectManager
Adding project 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\deploy-cromwell-on-azure\deploy-cromwell-on-azure.csproj'
info: OmniSharp.MSBuild.ProjectManager
Update project: deploy-cromwell-on-azure
info: OmniSharp.MSBuild.ProjectManager
Found referenced project outside root directory: c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\Common\Common.csproj
info: OmniSharp.MSBuild.ProjectManager
Adding project 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\Common\Common.csproj'
info: OmniSharp.MSBuild.ProjectManager
Queue project update for 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\Common\Common.csproj'
info: OmniSharp.MSBuild.ProjectManager
Loading project: c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\Common\Common.csproj
info: OmniSharp.MSBuild.ProjectManager
Successfully loaded project file 'c:\Users\v-blairmurri\source\repos\CromwellOnAzure\src\Common\Common.csproj'.
info: OmniSharp.MSBuild.ProjectManager
Update project: Common
info: OmniSharp.OmniSharpWorkspace
Adding analyzer reference: C:\Program Files\dotnet\sdk\5.0.402\Sdks\Microsoft.NET.Sdk\analyzers\Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll
info: OmniSharp.OmniSharpWorkspace
Adding analyzer reference: C:\Program Files\dotnet\sdk\5.0.402\Sdks\Microsoft.NET.Sdk\analyzers\Microsoft.CodeAnalysis.NetAnalyzers.dll
Received response for /quickinfo but could not find request.

C# log

Environment information

VSCode version: 1.61.2
C# Extension: 1.23.16

Dotnet Information

.NET SDK (reflecting any global.json): Version: 5.0.402 Commit: e9d3381880

Runtime Environment:
OS Name: Windows
OS Version: 10.0.22000
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\5.0.402\

Host (useful for support):
Version: 5.0.11
Commit: f431858f8b

.NET SDKs installed:
5.0.402 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

Extension	Author	Version
azps-tools	azps-tools	0.2.1
azure-account	ms-vscode	0.9.9
csharp	ms-dotnettools	1.23.16
vscode-azureresourcegroups	ms-azuretools	0.4.0
vscode-azurevirtualmachines	ms-azuretools	0.4.1
vscode-docker	ms-azuretools	1.17.0

[gregg-miskelly]

What is going on is that the paths in your PDB files are stamped with \remote\sources\path\Deployer.cs. So you need add or change your sourceFileMap entry to map \remote\sources\path to some path on the local computer. Documentation: https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md#source-file-map

[BMurri]

Actually, the PDBs are stamped with the equivalent of ${workspaceFolder}\Sourcefile.cs.

The debugger has to look in places like /remote/path/to/ or ~/Cromwell/code, neither path of which vscode has access to (basically, there is no path that vscode and the debugger share)

[gregg-miskelly]

That isn't a problem. The remote debugger never looks at source files. So you want sourceFileMap to contain entries that map "Path-in-PDB": "Path-where-VSCode-will-find-it".

[gregg-miskelly]

I tried to make things more clear: #4844

[BMurri]

I appreciate that. It better explains what should be happening, and much of what is happening.

Here's the weird thing: the PDBs are generated on the same machine as where VSCode is running, and are copied and are coresident with the executables where the debugger is running. So, that would suggest that my sourceFileMap needs to be the equivalent of ${workspaceFolder}: ${workspaceFolder}. Before I added sourceFileMap and copied my sources to the machine running the debugger, the debugger returned warnings for each breakpoint I set in VSCode to the effect that it could not locate the file at path "Path-where-VSCode-will-find-it" and never stopped at any of the breakpoints. By adding sourceFileMap entries that map Path-in-PDB: Path-where-Debugger-will-find-it and copying the sources to Path-where-Debugger-will-find-it the debugger now stops at the set breakpoints, but results in the issue that I opened.

So, I guess that my real issue is

Debugger does not stop at breakpoints set in VSCode if it cannot access source file at either Path-in-PDB or a projection of that path provided by sourceFileMap.

This explicit experience directly contradicts your statement (referring to source files):

It doesn't need to exist either on the computer running Visual Studio Code, or if you are remote debugging, on the remote machine.

[BMurri]

The basic problem is that the machine running the debugger does not have access to the repository nor does it have GUI access. Thus, builds run on the machine with VSCode, debugger runs on remote.

[gregg-miskelly]

I can't think of any reason why a sourceMap would be helpful in your scenario. You are sure that if you remove it, the breakpoints don't bind? Can you provide a log of what your setBreakpoints requests and responses look like in that case vs. the no sourceMap case?