Add support for automatically finding the macOS SDK

[PathogenDavid]

Discovered while investigating the effort required to stand up CI for macOS even though we don't officially support it yet.

Similar to #201, libclang-pathogen does not automatically discover the macOS SDK on its own. This causes basic system headers to not be found.

---

Apple Clang and Homebrew Clang do automagically "discover" the SDK.

Running clang -v for either reveals -isysroot is passed with a path to the SDK along with the resource directory, this (and maybe some of the other options near it) are probably what we're probably missing:

-resource-dir /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk
-I/usr/local/include
-stdlib=libc++
-internal-isystem /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1
-internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include
-internal-isystem /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include
-internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include
-internal-externc-isystem /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include

Manually specifying (non-cc1 equivalents of) these flags seems to fix the issue, so definitely a step in the right direction.

The -isysroot option is described as being for setting the system root directory. This seems odd, but the macOS SDK is structured like a Unix root with a ./usr/include and all that. (My understanding is prior to 10.15 macOS used to put include files in /usr/include but has transitioned to distributing them in Xcode like this.) Unfortunately this option alone is not enough to fix things.

---

I have found references to setting the SDKROOT environment variable to specify this SDK root, but this does not seem to work and I didn't find where it was handled within LLVM. This seems to be related to xcrun so maybe Clang uses it somewhere? (And that somewhere is inactive with libclang?) Nevermind, must've made a typo. It's handled in Darwin::AddDeploymentTarget. It being specified basically just implicitly adds -isysroot $SDKROOT.

---

I came across this PR from when they solved this issue for the Homebrew LLVM distribution: Homebrew/homebrew-core#45304 I'm not familiar with Homebrew formulas, but from what I can gather they just specify the preprocessor constant DEFAULT_SYSROOT with the path to the macOS SDK so it defaults to that.

According to the documentation for this constant, this is the equivalent of adding --sysroot to your invocations. (Looking at the code this appears to be accurate.) Unfortunately just like -isysroot this is not enough on its own.

---

As far as locating the SDK programmatically, from what I've found the canonical way is by running xcrun --sdk macosx --show-sdk-pathxcrun --show-sdk-path***. I believe what this returns is primarily controlled using the xcode-select, although there's some environment variables which can override it as well.

***Including --sdk macosx is likely a mistake. This actually ends up using the versionless SDK (IE: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk). Without it, my machine uses MacOSX10.15.sdk which feels more appropriate considering I'm running on 10.15. (MacOSX.sdk is a symlink to MacOSX11.1.sdk) I feel like this changed after installed Homebrew (and by extension - the Command Line Tools for Xcode), I seem to recall it used MacOSX.sdk regardless before.

(One thing that's not 100% clear to me is how things are handled when the SDK is provided by the command line tools app instead of Xcode. For example, on my MacBook the SDK is at /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk but on GitHub Actions it's at /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk. I don't think we really need to worry about this though.)

[PathogenDavid]

Since testing in CI was getting tedious I made a small app specifically for testing this issue on my Mac.

With -v --sysroot $(xcrun --sdk macosx --show-sdk-path) I can see the SDK path is only solving the issue for some of the paths. Here's a list of the system include paths for Apple Clang and whether they work with just --sysroot:

• /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1
• /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/include (Clang resource directory)
• /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
• /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
• /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)

The main one that's killing us is the first one. It's added by DarwinClang::AddClangCXXStdlibIncludeArgs:

// On Darwin, libc++ can be installed in one of the following two places:
// 1. Alongside the compiler in         <install>/include/c++/v1
// 2. In a SDK (or a custom sysroot) in <sysroot>/usr/include/c++/v1

With Apple Clang it is getting resolved by the first one, which doesn't apply to us since we aren't Apple Clang. Homebrew Clang uses that to resolve libc++ as well. Homebrew includes its own copy.

In theory we could include it too. However it is not currently built by our build scripts. I feel like the more appropriate thing to do here is to use the one in the platform SDK like we (implicitly) do for Windows and Linux.

[PathogenDavid]

I installed Homebrew which installed the Command Line Tools for Xcode and changed some other things. Apple Clang is provided by the command line tools instead now, so some paths changed:

• ✨/usr/local/include
• /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1 (libc++)
• ✨/Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/include (Clang resource directory)
  • This is the version provided by ClangSharp.Pathogen.ClangResources instead.
• /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include
• /Library/Developer/CommandLineTools/usr/include
• /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks (framework directory)

Paths marked with ✨ are present even without --sysroot. For some reason the first one actually goes away entirely when we pass --sysroot. (I thought this might've been a subtle different between --sysroot and -isysroot but this does not seem to be the case.)

Maybe worth noting: Without --sysroot Biohazrd ends up with two completely different framework directories. (/System/Library/Frameworks and /Library/Frameworks)

[PathogenDavid]

Back on the libc++ thing: I think the easiest solution for now is to just add a system include based on the location of the system Clang. We can find the system Clang using xcrun --find clang.

Same for the first one exception -I/usr/local/include. I suspect this one is only going away because --sysroot is messing with it.

With dotnet run --no-restore -- -v --sysroot $(xcrun --show-sdk-path) -I/usr/local/include -isystem /Library/Developer/CommandLineTools/usr/include/c++/v1 we now have:

• ✨/usr/local/include
• /Library/Developer/CommandLineTools/usr/bin/../include/c++/v1 (libc++)
• ✨/Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/include (Clang resource directory)
  • This is the version provided by ClangSharp.Pathogen.ClangResources instead.
• /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include
• /Library/Developer/CommandLineTools/usr/include
• /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks (framework directory)

That just leaves that non-SDK usr/include. I think this is getting added via C_INCLUDE_DIRS (it differs between Apple Clang and Xcode Clang): https://github.com/llvm/llvm-project/blob/2542c1a5a1306398d73c3c0c5d71cacf7c690093/clang/lib/Driver/ToolChains/Darwin.cpp#L2055-L2070 For both Apple Clang and Xcode Clang this is just the usr/include which contains libc++. -- Although we can't properly add it because --internal-externc-isystem is not easily accessible to us since it's a -cc1 option.

This one (along with /usr/local/include) are both missing from Homebrew Clang too, actually. Neither look especially important, but we'll probably keep them for consistency.

[PathogenDavid]

Putting this somewhere I'll end up finding it if it matters: I should poke around in https://opensource.apple.com/ to see if xcrun is in there. Quick search says no, but I do see it used in lots of build scripts so I think us using it this way is reasonable.

Also I forgot to note that I had a non-developer friend test xcrun on her Mac and the tool is present. It prints an error about there being no developer tools and returns exit code 1.

Unfortunately it also shows this dialog:

It's not clear if this dialog appears when xcrun has stdout redirected. It might be smart enough to detect it's being used non-interactively and skip that dialog. (Unfortunately said friend is on vacation now so I can't ask her to test.)