OSX PR build artifacts are missing symbols

[jkotas]

Repro

1. Download CoreCLRProduct__osx_x64 artifacts from PR build (e.g. https://dev.azure.com/dnceng-public/_apis/resources/Containers/40000142/CoreCLRProduct__osx_x64_release?itemPath=CoreCLRProduct__osx_x64_release%2FCoreCLRProduct__osx_x64_release.tar.gz)

Result

No *.dwarf or *.dbg files present in the archive.

This makes is impossible to debug OSX crash dumps created by PR test runs.

[dotnet-policy-service]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

[jkotas]

@BruceForstall Are you aware of this issue?

[BruceForstall]

@BruceForstall Are you aware of this issue?

I'm wasn't.

It looks like this behavior was introduced by #81387 (cc @kunalspathak @agocke ).

The build job, but only for PR builds, sets a flag to skip the "strip_symbols" step:

runtime/eng/pipelines/coreclr/templates/build-job.yml

Lines 101 to 105 in 40ac297

	- name: nativeSymbols
	value: ''
	- ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
	- name: nativeSymbols
	value: '--keepnativesymbols'

this ends up setting CLR_CMAKE_KEEP_NATIVE_SYMBOLS=true which causes symbol stripping to be skipped (eng/native/functions.cmake).

It appears this is ok for Linux, which apparently collect their debug symbols in the built executable. It is not ok for Mac, where the built executable only contains a link to the object files that contain the debug symbols. (The --keepsymbols option was specifically introduced to support Linux: #39203).

So, on Mac, we need to always "strip" symbols. "strip" is a bit of a misnomer in this case: it's more like "collect" symbols, to put them into a single .dwarf file (or, preferably, a .dSYM bundle).

A simple fix would be to not do pass --keepsymbols to the build-runtime.sh script for Mac

I presume the stack trace stuff has never worked on Mac?

Related: #92911

[hoyosjs]

We haven't notices because macOS queues have largely been unable to collect dumps until recently and even after that, the queues in Helix don't have the symbolizer. dotnet/arcade#11631

[janvorli]

the queues in Helix don't have the symbolizer

Instead of symbolizer, macOS has atos tool. An old note from my personal onenote has an example:

On macOS:
atos -o artifacts/bin/coreclr/OSX.arm64.Debug/libcoreclr.dylib.dwarf 0x7ac654
EEStartupHelper() (in libcoreclr.dylib.dwarf) (ceemain.cpp:1001)
(use the dwarf file to get the source line)

Or 
atos -o artifacts/bin/coreclr/OSX.arm64.Debug/libcoreclr.dylib.dwarf 0x7ac654 -fullPath
EEStartupHelper() (in libcoreclr.dylib.dwarf) (/Users/janvorli/git/runtime/src/coreclr/vm/ceemain.cpp:1001)

[hoyosjs]

They claim it's installed. I'd have to see if it works. Currently the infrastructure is such that it parses the output of that tool. Given this issue I expect it not to work since we don't have any logic to locate symbols. In the case of atos, it may be dsym-bundling aware which would help, but it's likely work. cc @JulieLeeMSFT