Specifying a SDK with -sdk /path has the driver looking in a wrong resource directory path too on non-Darwin

[finagolfin]

Description
I've started looking into moving the runtime libraries into the architecture-specific directory on Unix platforms, and I noticed this oddity in the paths it looks for libraries and the rpaths it applies. If I compile a simple executable on linux, here are the paths generated:

> ./swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swiftc swift/test/Interpreter/hello_toplevel.swift -v
Swift version 5.7.3 (swift-5.7.3-RELEASE)
Target: x86_64-unknown-linux-gnu
/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-frontend -frontend -c -primary-file swift/test/Interpreter/hello_toplevel.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -new-driver-path /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-driver -empty-abi-descriptor -resource-dir /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift -module-name hello_toplevel -o /tmp/TemporaryDirectory.ztzGDa/hello_toplevel-1.o
/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-autolink-extract /tmp/TemporaryDirectory.ztzGDa/hello_toplevel-1.o -o /tmp/TemporaryDirectory.ztzGDa/hello_toplevel-2.autolink
/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/clang -fuse-ld=gold -pie -Xlinker -rpath -Xlinker /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o /tmp/TemporaryDirectory.ztzGDa/hello_toplevel-1.o @/tmp/TemporaryDirectory.ztzGDa/hello_toplevel-2.autolink -L /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux -lswiftCore --target=x86_64-unknown-linux-gnu -v -o hello_toplevel
clang version 13.0.0 (https://github.com/apple/llvm-project.git c41f13252ed4b49f246729b4d91ff521d5a6bf9d)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/bin/ld.gold" -pie -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o hello_toplevel /lib/x86_64-linux-gnu/Scrt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginS.o -L/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/../lib -L/lib -L/usr/lib -rpath /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o /tmp/TemporaryDirectory.ztzGDa/hello_toplevel-1.o -lswift_StringProcessing -lswift_Concurrency -lswiftCore -lswiftSwiftOnoneSupport -lswiftCore -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/9/crtendS.o /lib/x86_64-linux-gnu/crtn.o

Note that it only links against -L /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux and adds a rpath for that one directory.

Steps to reproduce
However, if I make explicit the SDK it's using,

> ./swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swiftc swift/test/Interpreter/hello_toplevel.swift -sdk / -v
warning: Could not read SDKSettings.json for SDK at: /
Swift version 5.7.3 (swift-5.7.3-RELEASE)
Target: x86_64-unknown-linux-gnu
/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-frontend -frontend -c -primary-file swift/test/Interpreter/hello_toplevel.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -sdk / -new-driver-path /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-driver -empty-abi-descriptor -resource-dir /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift -module-name hello_toplevel -o /tmp/TemporaryDirectory.XikRsd/hello_toplevel-1.o
/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-autolink-extract /tmp/TemporaryDirectory.XikRsd/hello_toplevel-1.o -o /tmp/TemporaryDirectory.XikRsd/hello_toplevel-2.autolink
/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/clang -fuse-ld=gold -pie -Xlinker -rpath -Xlinker /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux -Xlinker -rpath -Xlinker /usr/lib/swift /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o /tmp/TemporaryDirectory.XikRsd/hello_toplevel-1.o @/tmp/TemporaryDirectory.XikRsd/hello_toplevel-2.autolink --sysroot / -L /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux -L /usr/lib/swift -lswiftCore --target=x86_64-unknown-linux-gnu -v -o hello_toplevel
clang version 13.0.0 (https://github.com/apple/llvm-project.git c41f13252ed4b49f246729b4d91ff521d5a6bf9d)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin
Found candidate GCC installation: //lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: //lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/bin/ld.gold" --sysroot=/ -pie -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o hello_toplevel //lib/x86_64-linux-gnu/Scrt1.o //lib/x86_64-linux-gnu/crti.o //lib/gcc/x86_64-linux-gnu/9/crtbeginS.o -L/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux -L/usr/lib/swift -L//lib/gcc/x86_64-linux-gnu/9 -L//lib/gcc/x86_64-linux-gnu/9/../../../../lib64 -L//lib/x86_64-linux-gnu -L//lib/../lib64 -L//usr/lib/x86_64-linux-gnu -L//usr/lib/../lib64 -L/home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/../lib -L//lib -L//usr/lib -rpath /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux -rpath /usr/lib/swift /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift/linux/x86_64/swiftrt.o /tmp/TemporaryDirectory.XikRsd/hello_toplevel-1.o -lswift_StringProcessing -lswift_Concurrency -lswiftCore -lswiftSwiftOnoneSupport -lswiftCore -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed //lib/gcc/x86_64-linux-gnu/9/crtendS.o //lib/x86_64-linux-gnu/crtn.o

now it's also looking in -L /usr/lib/swift and adding that rpath too, when I only want it to look in the local toolchain. This happens even if I specify the resource directory with -resource-dir /home/butta/swift-5.7.3-RELEASE-ubuntu20.04/usr/lib/swift in addition to the SDK.

Expected behavior
While the default behavior is not to specify the SDK or resource directory and assume they are at / and /usr/lib/swift, one often specifies both when cross-compiling, ie a C cross-compilation sysroot is the SDK and the resource directory is a separate Swift directory. In that case, the compiler looks in a separate extra directory in the C sysroot where there are no Swift files. While this may not cause problems because that directory either doesn't exist or has no libraries, we should clean that up. @colemancda, could you confirm that you see this issue with your linux cross-compilation sysroots?

Environment

• Swift 5.7.3 for Ubuntu 20.04 x86_64

Additional context
This is because of this block added by @beccadax in 2019, which was most likely a mistake as it doesn't add the platform name, ie it adds /usr/lib/swift not the correct /usr/lib/swift/linux. It was most likely incorrectly added to get the resource directory from the SDK, but since it didn't work, @compnerd soon after added a working method in #25990.

I'd like to submit a pull to remove or disable that last block from getRuntimeLibraryPaths(), but I'm unsure if other platforms rely on it, ie /path/to/sdk/usr/lib/swift. Linux definitely doesn't, putting the runtime libraries in /path/to/sdk/usr/lib/swift/linux instead, but I've never used the non-Unix toolchains.

Could @beccadax, @compnerd, and @MaxDesiatov let me know if you need this block for Darwin, Windows, or Wasm, and I'll submit a pull to disable it for the platforms where it's unused?

[finagolfin]

@AnthonyLatsis, this isn't really related to cross-compilation, would you update the tags? I have a fix submitted that should be merged soon.