SourceMaps stopped working for bazel-built processes on version 1.11.2 and 1.11.3

[dpar39]

OS: Ubuntu
VSCode version: 1.96.4
CodeLLDB version: 1.11.2
Compiler: clang 14
Debuggee: ELF x64 binaries on Linux with glibc 2.35

When debugging our bazel targets, in version 1.11.2 (or 1.11.3), our sourceMap from "/proc/self/cwd" to "${workspaceFolder:code_root_folder}" stopped working meaning that breakpoints are no longer hit. The same configuration works with version 1.11.1 and older. Seems some regression was introduced in 1.11.2, but I can't quite put my finger on it looking at v1.11.1...v1.11.2

UPDATE: It seems to affect only requests of type "attach". "launch" configurations are not affected by the regression.

[dpar39]

Here's my bet on where the problem could be: v1.11.1...v1.11.2#diff-14b183725ee1b7204a41e6600c42c555845df6c5e7ead8bc4705c96c26bc4f90R278

[dpar39]

Changing the lldb library didn't make a difference, so it points the problem is somewhere in the adapter.
1.11.1 with lldb.library pointing to ~/.vscode-server/extensions/vadimcn.vscode-lldb-1.11.2/lldb/lib/liblldb.so works, but 1.11.2 pointing to ~/.vscode-server/extensions/vadimcn.vscode-lldb-1.11.1/lldb/lib/liblldb.so does not!

I also tried dumping the merged object in mergeWorkspaceSettings() and the sourceMap looks correct. Adding it to "lldb.launch.sourceMap" didn't make a difference either as expected.

[vadimcn]

... our sourceMap from "/proc/self/cwd" to "${workspaceFolder:code_root_folder}" ...

I'm not familiar with this syntax. Did you mean ${workspaceFolder}/code_root_folder?

Try dumping the effective source map via settings show target.source-map.
Also, seeing a verbose log would be helpful...

[dpar39]

setttings show target.source-map shows empty in the debugger console using 1.11.2 or 1.11.3:

target.source-map (path-map) =

in 1.11.1 I get (somewhat redacted):

target.source-map (path-map) =
[0] "/proc/self/cwd" -> "/home/user/project"

Here's some more information. Adding settings set target.source-map /proc/self/cwd /home/user/project to my init commands did not work either nor it it worked adding it in the debugger console once the code stopped on a SIGABRT.

Regarding the "${workspaceFolder:<folder-name>}", this is used to refer to a specific folder in a multi-root workspace. Read https://code.visualstudio.com/docs/editor/variables-reference#_variables-scoped-per-workspace-folder for more information.

[dpar39]

Regarding the logging information, here's the relevant part (again with some paths redacted) which shows that it can't resolve breakpoints:

Platform: linux x64
Initial debug configuration: {
  name: 'python lldb attach',
  type: 'lldb',
  request: 'attach',
  pathMappings: null,
  connect: null,
  sourceMap: { '/proc/self/cwd': '${workspaceFolder:code_root_folder}' },
  pid: '306427',
  initCommands: [
    'settings set target.process.stop-on-exec false',
    'settings set target.process.follow-fork-mode child',
    'pro hand -p true -s false SIGWINCH'
  ]
}
Resolved debug configuration: {
  name: 'python lldb attach',
  type: 'lldb',
  request: 'attach',
  pathMappings: null,
  connect: null,
  sourceMap: { '/proc/self/cwd': '${workspaceFolder:code_root_folder}' },
  pid: '306427',
  initCommands: [
    'settings set target.process.stop-on-exec false',
    'settings set target.process.follow-fork-mode child',
    'pro hand -p true -s false SIGWINCH'
  ],
  relativePathBase: '/home/user/project',
  _adapterSettings: {
    displayFormat: 'auto',
    showDisassembly: 'auto',
    dereferencePointers: true,
    suppressMissingSourceFiles: true,
    evaluationTimeout: 5,
    consoleMode: 'commands',
    sourceLanguages: null,
    scriptConfig: {},
    terminalPromptClear: null,
    evaluateForHovers: true,
    commandCompletions: true,
    reproducer: false
  }
}
Launching adapter
liblldb: /home/user/.vscode-server/extensions/vadimcn.vscode-lldb-1.11.3/lldb/lib/liblldb.so
environment: {}
settings: { evaluateForHovers: true, commandCompletions: true }
[DEBUG codelldb] Connecting to 127.0.0.1:39969
[DEBUG codelldb] New debug session
[DEBUG codelldb::dap_codec] --> {"command":"initialize","arguments":{"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"lldb","pathFormat":"path","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"locale":"en","supportsProgressReporting":true,"supportsInvalidatedEvent":true,"supportsMemoryReferences":true,"supportsArgsCanBeInterpretedByShell":true,"supportsMemoryEvent":true,"supportsStartDebuggingRequest":true,"supportsANSIStyling":true},"type":"request","seq":1}
[DEBUG codelldb::dap_codec] <-- {"seq":1,"type":"response","request_seq":1,"success":true,"command":"initialize","body":{"exceptionBreakpointFilters":[{"default":true,"filter":"cpp_throw","label":"C++: on throw","supportsCondition":true},{"default":false,"filter":"cpp_catch","label":"C++: on catch","supportsCondition":true}],"supportTerminateDebuggee":true,"supportsCancelRequest":true,"supportsClipboardContext":true,"supportsCompletionsRequest":true,"supportsConditionalBreakpoints":true,"supportsConfigurationDoneRequest":true,"supportsDataBreakpointBytes":true,"supportsDataBreakpoints":true,"supportsDelayedStackTraceLoading":true,"supportsDisassembleRequest":true,"supportsEvaluateForHovers":true,"supportsExceptionFilterOptions":true,"supportsExceptionInfoRequest":true,"supportsFunctionBreakpoints":true,"supportsGotoTargetsRequest":true,"supportsHitConditionalBreakpoints":true,"supportsInstructionBreakpoints":true,"supportsLogPoints":true,"supportsModulesRequest":true,"supportsReadMemoryRequest":true,"supportsRestartRequest":true,"supportsSetVariable":true,"supportsStepInTargetsRequest":true,"supportsSteppingGranularity":true,"supportsWriteMemoryRequest":true}}
[DEBUG codelldb::dap_codec] --> {"command":"attach","arguments":{"name":"python lldb attach","type":"lldb","request":"attach","pathMappings":null,"connect":null,"sourceMap":{"/proc/self/cwd":"/home/user/project"},"pid":"306427","initCommands":["settings set target.process.stop-on-exec false","settings set target.process.follow-fork-mode child","pro hand -p true -s false SIGWINCH"],"relativePathBase":"/home/user/project","_adapterSettings":{"displayFormat":"auto","showDisassembly":"auto","dereferencePointers":true,"suppressMissingSourceFiles":true,"evaluationTimeout":5,"consoleMode":"commands","sourceLanguages":null,"scriptConfig":{},"terminalPromptClear":null,"evaluateForHovers":true,"commandCompletions":true,"reproducer":false},"__sessionId":"3ede3118-6a6c-4736-9ded-be6c11415076"},"type":"request","seq":2}
[INFO  codelldb::debug_session::launch] Set target.source-map args: "/proc/self/cwd" "/home/user/project" 
[DEBUG codelldb::dap_codec] <-- {"seq":2,"type":"event","event":"output","body":{"category":"console","output":"Console is in 'commands' mode, prefix expressions with '?'.\n"}}
[DEBUG codelldb::debug_session] settings set target.process.stop-on-exec false -> SuccessFinishResult, Error:  Success
[DEBUG codelldb::dap_codec] <-- {"seq":3,"type":"event","event":"output","body":{"category":"console","output":"Executing script: initCommands\n"}}
[DEBUG codelldb::debug_session] settings set target.process.follow-fork-mode child -> SuccessFinishResult, Error:  Success
[DEBUG codelldb::debug_session] pro hand -p true -s false SIGWINCH -> SuccessFinishResult, Error:  Success
    Output Message:
    NAME         PASS     STOP     NOTIFY
    ===========  =======  =======  =======
    SIGWINCH     true     false    not set
    
[DEBUG codelldb::dap_codec] <-- {"seq":4,"type":"event","event":"output","body":{"category":"console","output":"NAME         PASS     STOP     NOTIFY\n===========  =======  =======  =======\nSIGWINCH     true     false    not set\n\n"}}
INFO(Python) 08:52:24 lang_support: languages: set()
[DEBUG codelldb::dap_codec] <-- {"seq":5,"type":"event","event":"initialized"}
[DEBUG codelldb::dap_codec] --> {"command":"setBreakpoints","arguments":{"source":{"name":"somefile.cc","path":"/home/user/project/somefile.cc"},"lines":[27],"breakpoints":[{"line":27}],"sourceModified":false},"type":"request","seq":3}
[DEBUG codelldb::dap_codec] <-- {"seq":6,"type":"response","request_seq":3,"success":true,"command":"setBreakpoints","body":{"breakpoints":[{"id":1,"message":"Resolved locations: 0","verified":false}]}}
[DEBUG codelldb::dap_codec] --> {"command":"setFunctionBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":4}
[DEBUG codelldb::dap_codec] --> {"command":"setInstructionBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":5}
[DEBUG codelldb::dap_codec] --> {"command":"setExceptionBreakpoints","arguments":{"filters":[],"filterOptions":[]},"type":"request","seq":6}
[DEBUG codelldb::dap_codec] --> {"command":"setDataBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":7}
[DEBUG codelldb::dap_codec] <-- {"seq":7,"type":"response","request_seq":4,"success":true,"command":"setFunctionBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] <-- {"seq":8,"type":"response","request_seq":5,"success":true,"command":"setInstructionBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] <-- {"seq":9,"type":"response","request_seq":6,"success":true,"command":"setExceptionBreakpoints"}
[DEBUG codelldb::dap_codec] <-- {"seq":10,"type":"response","request_seq":7,"success":true,"command":"setDataBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] --> {"command":"configurationDone","type":"request","seq":8}
[DEBUG codelldb::dap_codec] <-- {"seq":11,"type":"event","event":"output","body":{"category":"console","output":"Attached to process 306427\n"}}
[DEBUG codelldb::dap_codec] <-- {"seq":12,"type":"response","request_seq":2,"success":true,"command":"attach"}
[DEBUG codelldb::debug_session] Debug event: 0x7f9cd008f990 Event: broadcaster = 0x555cffb541a8 (lldb.process), type = 0x00000001 (state-changed), data = { process = 0x555cffb54170 (pid = 306427), state = running}
[DEBUG codelldb::dap_codec] <-- {"seq":13,"type":"response","request_seq":8,"success":true,"command":"configurationDone"}
[DEBUG codelldb::dap_codec] <-- {"seq":14,"type":"event","event":"continued","body":{"allThreadsContinued":true,"threadId":306427}}
[DEBUG codelldb::dap_codec] --> {"command":"threads","type":"request","seq":9}
[DEBUG codelldb::dap_codec] <-- {"seq":15,"type":"response","request_seq":9,"success":true,"command":"threads","body":{"threads":[{"id":306427,"name":"1: tid=306427"},{"id":306483,"name":"2: tid=306483"},{"id":306484,"name":"3: tid=306484"},{"id":306486,"name":"4: tid=306486"},{"id":306487,"name":"5: tid=306487"}]}}

[vadimcn]

I can't reproduce your problem. The following config works for me:

{
	"name": "Attach by pid",
	"type": "lldb",
	"request": "attach",
	"sourceMap": { "/proc/self/cwd": "${workspaceFolder}" },
	"pid": 2515521
}

After attaching the source map is set as expected.

Can you try a similar simple config on your end?

[Penguinang]

I find this problem on v1.11.3 too: source-map in settings.json is not working(after launching, I typed 'settings show target.source-map' dumps empty).
After downgrading to v1.11.2, the problem does reproduce.
After downgrading to v1.11.1, the problem does not reproduce.
And I'm not using bazel

[vadimcn]

@Penguinang can you share your launch configuration? Or, better yet, a verbose log?

[evg-n]

Hello @vadimcn , seems like I'm and a bunch of my colleagues experiencing the same issue. I'm using bazel, from some point in time source maps (when I'm attaching to the process) just stopped working properly. I'm on 1.11.3 but downgrading didn't help.

Maybe the issue is somewhere in tools or libraries that your extension is using in vscode, not in extension code itself, because it seems like the same thing (broken source maps for debug build) actually started to happen with GDB debugger. Appreciate your work anyways.

This is what I see when I'm trying to put a breakpoint somewhere in my source code:
Breakpoint at /home/evgnpt/.../my_repo/my_cpp_module.cpp:162 could not be resolved, but a valid location was found at /proc/self/cwd/.../my_repo/my_cpp_module.cpp:162

I'm using remote (ubuntu) dev containers in vscode. Here is verbose log:

Verbose log

Verbose logging: on  (Use "lldb.verboseLogging" setting to change)
Platform: linux x64
Initial debug configuration: {
  type: 'lldb',
  request: 'attach',
  name: 'LLDB Attach',
  pid: '${command:pickProcess}',
  sourceMap: { '/proc/self/cwd': '${workspaceFolder}' },
  __configurationTarget: 6
}
Resolved debug configuration: {
  type: 'lldb',
  request: 'attach',
  name: 'LLDB Attach',
  pid: '${command:pickProcess}',
  sourceMap: { '/proc/self/cwd': '${workspaceFolder}' },
  __configurationTarget: 6,
  relativePathBase: '${workspaceFolder}',
  _adapterSettings: {
    displayFormat: 'auto',
    showDisassembly: 'always',
    dereferencePointers: true,
    suppressMissingSourceFiles: true,
    evaluationTimeout: 5,
    consoleMode: 'commands',
    sourceLanguages: null,
    scriptConfig: {},
    terminalPromptClear: null,
    evaluateForHovers: true,
    commandCompletions: true,
    reproducer: false
  }
}
Launching adapter
liblldb: /home/evgnpt/.vscode-server/extensions/vadimcn.vscode-lldb-1.11.3/lldb/lib/liblldb.so
environment: {}
settings: { evaluateForHovers: true, commandCompletions: true }
[DEBUG codelldb] Connecting to 127.0.0.1:46521
[DEBUG codelldb] New debug session
[DEBUG codelldb::dap_codec] --> {"command":"initialize","arguments":{"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"lldb","pathFormat":"path","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"locale":"en","supportsProgressReporting":true,"supportsInvalidatedEvent":true,"supportsMemoryReferences":true,"supportsArgsCanBeInterpretedByShell":true,"supportsMemoryEvent":true,"supportsStartDebuggingRequest":true,"supportsANSIStyling":true},"type":"request","seq":1}
[DEBUG codelldb::dap_codec] <-- {"seq":1,"type":"response","request_seq":1,"success":true,"command":"initialize","body":{"exceptionBreakpointFilters":[{"default":true,"filter":"cpp_throw","label":"C++: on throw","supportsCondition":true},{"default":false,"filter":"cpp_catch","label":"C++: on catch","supportsCondition":true}],"supportTerminateDebuggee":true,"supportsCancelRequest":true,"supportsClipboardContext":true,"supportsCompletionsRequest":true,"supportsConditionalBreakpoints":true,"supportsConfigurationDoneRequest":true,"supportsDataBreakpointBytes":true,"supportsDataBreakpoints":true,"supportsDelayedStackTraceLoading":true,"supportsDisassembleRequest":true,"supportsEvaluateForHovers":true,"supportsExceptionFilterOptions":true,"supportsExceptionInfoRequest":true,"supportsFunctionBreakpoints":true,"supportsGotoTargetsRequest":true,"supportsHitConditionalBreakpoints":true,"supportsInstructionBreakpoints":true,"supportsLogPoints":true,"supportsModulesRequest":true,"supportsReadMemoryRequest":true,"supportsRestartRequest":true,"supportsSetVariable":true,"supportsStepInTargetsRequest":true,"supportsSteppingGranularity":true,"supportsWriteMemoryRequest":true}}
[DEBUG codelldb::dap_codec] --> {"command":"attach","arguments":{"type":"lldb","request":"attach","name":"LLDB Attach","pid":"154070","sourceMap":{"/proc/self/cwd":"...$workspace..."},"__configurationTarget":6,"relativePathBase":"...$workspace...","_adapterSettings":{"displayFormat":"auto","showDisassembly":"always","dereferencePointers":true,"suppressMissingSourceFiles":true,"evaluationTimeout":5,"consoleMode":"commands","sourceLanguages":null,"scriptConfig":{},"terminalPromptClear":null,"evaluateForHovers":true,"commandCompletions":true,"reproducer":false},"__sessionId":"6c843241-4910-4534-aff3-fea6a3a834a6"},"type":"request","seq":2}
[INFO  codelldb::debug_session::launch] Set target.source-map args: "/proc/self/cwd" "...$workspace..." 
[DEBUG codelldb::dap_codec] <-- {"seq":2,"type":"event","event":"output","body":{"category":"console","output":"Console is in 'commands' mode, prefix expressions with '?'.\n"}}
INFO(Python) 17:55:44 lang_support: languages: set()
[DEBUG codelldb::dap_codec] <-- {"seq":3,"type":"event","event":"initialized"}
[DEBUG codelldb::dap_codec] --> {"command":"setBreakpoints","arguments":{"source":{"name":"my_cpp_module.cpp","path":"...some_cpp_module_path..."},"lines":[60,151,154,162],"breakpoints":[{"line":60},{"line":151},{"line":154},{"line":162}],"sourceModified":false},"type":"request","seq":3}
[DEBUG codelldb::dap_codec] --> {"command":"setBreakpoints","arguments":{"source":{"name":"ride_prefetch_sender.cpp","path":"...some_cpp_module_path..."},"lines":[],"breakpoints":[],"sourceModified":false},"type":"request","seq":4}
[DEBUG codelldb::dap_codec] --> {"command":"setBreakpoints","arguments":{"source":{"name":"test_sttr_utils.cpp","path":"...some_cpp_module_path..."},"lines":[],"breakpoints":[],"sourceModified":false},"type":"request","seq":5}
[DEBUG codelldb::dap_codec] --> {"command":"setFunctionBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":6}
[DEBUG codelldb::dap_codec] --> {"command":"setInstructionBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":7}
[DEBUG codelldb::dap_codec] --> {"command":"setExceptionBreakpoints","arguments":{"filters":[],"filterOptions":[{"filterId":"cpp_catch"}]},"type":"request","seq":8}
[DEBUG codelldb::dap_codec] --> {"command":"setDataBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":9}
[DEBUG codelldb::dap_codec] <-- {"seq":4,"type":"response","request_seq":3,"success":true,"command":"setBreakpoints","body":{"breakpoints":[{"id":1,"message":"Resolved locations: 0","verified":false},{"id":3,"message":"Resolved locations: 0","verified":false},{"id":5,"message":"Resolved locations: 0","verified":false},{"id":7,"message":"Resolved locations: 0","verified":false}]}}
[DEBUG codelldb::dap_codec] <-- {"seq":5,"type":"response","request_seq":4,"success":true,"command":"setBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] <-- {"seq":6,"type":"response","request_seq":5,"success":true,"command":"setBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] <-- {"seq":7,"type":"response","request_seq":6,"success":true,"command":"setFunctionBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] <-- {"seq":8,"type":"response","request_seq":7,"success":true,"command":"setInstructionBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] <-- {"seq":9,"type":"response","request_seq":8,"success":true,"command":"setExceptionBreakpoints"}
[DEBUG codelldb::dap_codec] <-- {"seq":10,"type":"response","request_seq":9,"success":true,"command":"setDataBreakpoints","body":{"breakpoints":[]}}
[DEBUG codelldb::dap_codec] --> {"command":"configurationDone","type":"request","seq":10}
[DEBUG codelldb::dap_codec] <-- {"seq":11,"type":"event","event":"output","body":{"category":"console","output":"Attached to process 154070\n"}}
[DEBUG codelldb::dap_codec] <-- {"seq":12,"type":"response","request_seq":2,"success":true,"command":"attach"}
[DEBUG codelldb::debug_session] Debug event: 0x7f87ac089b00 Event: broadcaster = 0x56214eb5c3a8 (lldb.process), type = 0x00000001 (state-changed), data = { process = 0x56214eb5c370 (pid = 154070), state = running}
[DEBUG codelldb::dap_codec] <-- {"seq":13,"type":"response","request_seq":10,"success":true,"command":"configurationDone"}
[DEBUG codelldb::dap_codec] <-- {"seq":14,"type":"event","event":"continued","body":{"allThreadsContinued":true,"threadId":154070}}
[DEBUG codelldb::dap_codec] --> {"command":"threads","type":"request","seq":11}
[DEBUG codelldb::dap_codec] <-- {"seq":15,"type":"response","request_seq":11,"success":true,"command":"threads","body":{"threads":[{"id":154070,"name":"1: tid=154070"},{"id":154095,"name":"2: tid=154095"},{"id":154097,"name":"3: tid=154097"},{"id":154098,"name":"4: tid=154098"},{"id":154101,"name":"5: tid=154101"},{"id":154102,"name":"6: tid=154102"},{"id":154254,"name":"7: tid=154254"},{"id":154255,"name":"8: tid=154255"},{"id":154256,"name":"9: tid=154256"},{"id":154257,"name":"10: tid=154257"},{"id":154258,"name":"11: tid=154258"},{"id":154358,"name":"12: tid=154358"}]}}

upd: native vscode gdb extension seems to be working fine when I'm attaching to the process, source-maps are loading correctly.

[vadimcn]

I think I am starting to get the hang of what's happening: the source map is missing in cases when the attach happens without specifying the image location via program. In this case debugging target is supposed to get created dynamically; however, clearly something is broken.
For now, I recommend adding "program": "path/to/the/debuggee" into your launch configs.

[vadimcn]

Fixed in v1.11.4

[dpar39]

Super! Thanks so much! I can confirm that the new release doesn't have the bug anymore.