cross compilation support (configurable destinations)

[weissi]

General

This adds experimental support for cross compilation or more precisely configurable destinations for binaries to SwiftPM. The basic idea is that you hand a JSON file in the format described below to SwiftPM (ie. swift build --destination my-destination.json). SwiftPM will then invoke the swiftc/clang with the appropriate arguments. The --destination switch is optional and if omitted SwiftPM will use the destination as appropriate for the host OS as it previously did.

destination.json File Format

{
    "version": 1,
    "sdk": <path to the SDK for the target platform: String>,
    "toolchain-bin-dir": <path to the toolchain's bin dir: String>,
    "target": <LLVM target triple: String>,
    "extra-cc-flags": <extra C compiler arguments: [String]>,
    "extra-swiftc-flags": <extra Swift compiler arguments: [String]>,
    "dynamic-library-extension": <extension of dynamic library files: String>,
    "extra-cpp-flags": <extra C++ compiler flags: [String]>
}

Example: compile for macOS on macOS

{
    "version": 1,
    "sdk": "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/",
    "toolchain-bin-dir": "/Library/Developer/Toolchains/swift-3.1-RELEASE.xctoolchain/usr/bin",
    "target": "x86_64-apple-macosx10.10",
    "extra-cc-flags": [
        "-arch", "x86_64",
        "-mmacosx-version-min=10.10"
    ],
    "extra-swiftc-flags": [],
    "extra-cpp-flags": ["-lc++"],
    "dynamic-library-extension": "dylib"
}

Example: compile for Ubuntu Linux on macOS (SDK required)

{
    "version": 1,
    "sdk": "/Users/johannes/cross-toolchain/ubuntu-xenial.sdk",
    "toolchain-bin-dir": "/Users/johannes/cross-toolchain/swift.xctoolchain/usr/bin",
    "target": "linux-unknown-x86_64",
    "dynamic-library-extension": "so",
    "extra-cc-flags": [
        "-fPIC"
    ],
    "extra-swiftc-flags": [
        "-use-ld=gold"
    ],
    "extra-cpp-flags": ["-lstdc++"]
}

Some Words About Cross Compilation

To cross compile a binary, you will need a cross compilation toolchain which consists of the toolchain itself and an SDK. For example compiling an iPhone binary from your Mac will use /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.3.sdk or similar. If you want to compile a Swift binary for say Ubuntu Xenial on your Mac, you'll need the following pieces of software

• a compiler being able to cross compile ✅ (swiftc and clang are able to do that)
• the basic Swift libraries (like libswiftCore) for Ubuntu ✅ (that's part of the Swift distribution for Ubuntu)
• an SDK for Ubuntu Linux ✅ (that can be assembled from the .deb packages available from Ubuntu)
• an ELF linker that runs on macOS ✅ (the gold linker from the GNU binutils can be compiled on macOS)

In other words, all the pieces of software are available for free online, they just need to be assembled in the correct way. An example script to do that all for Ubuntu Xenial can be found in a separate PR.

[ddunbar]

@swift-ci please smoke test

[aciidgh]

This method needs doc comments and tests. Can you propose this as a separate change?

[weissi]

done (it's a separate commit so far, @aciidb0mb3r want me to make a separate PR?)

[aciidgh]

Separate PR would be nice but commit sounds fine too!

[weissi]

here you go: #1100

[aciidgh]

Please add doc comments.

[weissi]

done

[aciidgh]

Unless we want this to be hidden option, we should add help text.

[weissi]

I thought we'd add it as an experimental feature first but happy to add text. @aciidb0mb3r / @ddunbar any opinions?

[aciidgh]

Ah, then no help text needed.

[aciidgh]

Is this needed for this patch?

[weissi]

no, removed

[aciidgh]

This shouldn't be changed.

[weissi]

done

[aciidgh]

This looks like something which should be a part of toolchain and not each target and product.

[weissi]

@aciidb0mb3r the problem is that Product requires the shared library extension which is calculated from the Destination. So I just wired the Destination through everything. Prefer me to wire through the extension only or the UserToolchain?

[aciidgh]

Ah, for outname property? I think we should just lift that into the Build module. Doing that should allow keeping the destination completely inside the toolchain.

[weissi]

done

[aciidgh]

We can use let, right?

[weissi]

yes, but it's a value type so changes only affect the very local variable. I'm happy to make it let but don't think it makes much difference because it's a struct. Let me know what you think.

[aciidgh]

I think its better to start with let unless you really need var because it is part of the API. If something is let, clients won't have a question "what happens if I change the value". It also helps in case you need to convert the struct to a class.

[weissi]

cool, changed.

[aciidgh]

I think my major feedback is, moving the destination to be part of the UserToolchain and not have it per target and product. Also, please add some brief Swift doc comments where applicable.

[weissi]

thanks very much @aciidb0mb3r for your review. Will work on the bigger items tomorrow, fixed some smaller ones.

[weissi]

hey @aciidb0mb3r I now moved Destination from the PackageModel module into the Build module and took the outname out into the Build module too. Is this roughly what you were thinking?

[weissi]

@aciidb0mb3r I also added a new property "extra-cpp-flags" for flags for C++, I discovered earlier that that was still handled through #if os(macOS) which is bad...

[aciidgh]

The package builder is the convention system that loads a package from disk, it doesn't need to care about the output file name. We should move the computation of output path/name into the class ProductBuildDescription (It is inside BuildPlan.swift file).

[aciidgh]

Please avoid the diffs that do not contribute any functionality.

[weissi]

sorry about that, didn't spot those, removed.

[aciidgh]

These diffs are not required, right?

[weissi]

sorry about that, didn't spot those, removed.

[aciidgh]

As I commented above, we don't need to have the outname inside the target. It can be a build time thing.

[aciidgh]

We do these special things here to run tests inside Xcode, please ensure the tests work inside Xcode too by generating a project using Utilities/bootstrap --generate-xcodeproj

[aciidgh]

Please add a short description about this structure.

[weissi]

done

[aciidgh]

Please add a doc comment for each property

[weissi]

done

[aciidgh]

Please avoid abbreviated variable names. We try to follow Swift design guidelines https://swift.org/documentation/api-design-guidelines/

[weissi]

done

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[aciidgh]

@swift-ci please smoke test

[sebsto]

If you happened, like me, to find this because you search a way to cross compile for x86_64 and arm using the swift compiler. There is now (3-4 years later) a much easier solution. Unfortunately, it is not documented by swift build -h

echo "🏗 Building the ARM version"
swift build --configuration release \
            --arch arm64

echo "🏗 Building the x86_64 version"
swift build --configuration release \
            --arch x86_64

I hope it helps

[keith]

You can at least do it in a single invocation https://www.smileykeith.com/2020/12/24/swiftpm-cross-compile/