Build errors with nested dependencies and --use-xcframeworks #3122

EmDee · 2021-02-04T19:18:24Z

carthage install method: [ ] .pkg, [x] homebrew, [ ] source
which carthage: /usr/local/bin/carthage
carthage version: 0.37.0
xcodebuild -version: Xcode 12.4, Build version 12D4e
Are you using --no-build? No
Are you using --no-use-binaries? Yes
Are you using --use-submodules? No
Are you using --cache-builds? Yes
Are you using --new-resolver? No

Cartfile

github "pace/cloud-sdk-ios" "9f6ba1a79208e86bca0b27992b8febaccb68abcd"

Cartfile.resolved

github "apple/swift-protobuf" "1.15.0"
github "mattrubin/Base32" "1.1.2+xcode10.2"
github "mattrubin/OneTimePassword" "3.2.0"
github "openid/AppAuth-iOS" "e191b1beadf3041259652e6e464eaceff16e5fbb"
github "pace/cloud-sdk-ios" "9f6ba1a79208e86bca0b27992b8febaccb68abcd"

The command I ran: carthage bootstrap --platform iOS --cache-builds --no-use-binaries --use-xcframeworks

Carthage Output

   carthage bootstrap --platform iOS --cache-builds --no-use-binaries --use-xcframeworks
*** No Cartfile.resolved found, updating dependencies
*** Cloning cloud-sdk-ios
*** Cloning swift-protobuf
*** Cloning AppAuth-iOS
*** Cloning OneTimePassword
*** Cloning Base32
*** Checking out OneTimePassword at "3.2.0"
*** Checking out Base32 at "1.1.2+xcode10.2"
*** Checking out AppAuth-iOS at "e191b1beadf3041259652e6e464eaceff16e5fbb"
*** Checking out swift-protobuf at "1.15.0"
*** Checking out cloud-sdk-ios at "9f6ba1a79208e86bca0b27992b8febaccb68abcd"
*** No cache found for AppAuth-iOS, building with all downstream dependencies
*** No cache found for Base32, building with all downstream dependencies
*** No cache found for swift-protobuf, building with all downstream dependencies
*** xcodebuild output can be found in /var/folders/tr/rls_gynd5wd4hzq49f0ghq700000gn/T/carthage-xcodebuild.fNUZEP.log
*** Building scheme "AppAuth_iOS" in AppAuth.xcodeproj
*** Building scheme "AppAuthEnterpriseUserAgent" in AppAuth.xcodeproj
*** Building scheme "AppAuthCore" in AppAuth.xcodeproj
*** Building scheme "Base32 (iOS)" in Base32.xcodeproj
*** Building scheme "OneTimePassword (iOS)" in OneTimePassword.xcworkspace
*** Building scheme "SwiftProtobuf_iOS" in SwiftProtobuf.xcodeproj
*** Building scheme "PACECloudSDK" in PACECloudSDK.xcodeproj
Build Failed
	Task failed with exit code 65:
	/usr/bin/xcrun xcodebuild -project /Users/martin/Carthage/Checkouts/cloud-sdk-ios/PACECloudSDK.xcodeproj -scheme PACECloudSDK -configuration Release -derivedDataPath /Users/martin/Library/Caches/org.carthage.CarthageKit/DerivedData/12.4_12D4e/cloud-sdk-ios/9f6ba1a79208e86bca0b27992b8febaccb68abcd -sdk iphoneos ONLY_ACTIVE_ARCH=NO CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES archive VALIDATE_WORKSPACE=NO -archivePath /var/folders/tr/rls_gynd5wd4hzq49f0ghq700000gn/T/cloud-sdk-ios SKIP_INSTALL=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=NO CLANG_ENABLE_CODE_COVERAGE=NO STRIP_INSTALLED_PRODUCT=NO (launched in /Users/martin/Carthage/Checkouts/cloud-sdk-ios)

This usually indicates that project itself failed to compile. Please check the xcodebuild log for more details: /var/folders/tr/rls_gynd5wd4hzq49f0ghq700000gn/T/carthage-xcodebuild.fNUZEP.log

The last lines of the log files:

no such module 'AppAuth'
import AppAuth
               ^
** ARCHIVE FAILED **


The following build commands failed:
  CompileSwift normal arm64
  CompileSwiftSources normal arm64 com.apple.xcode.tools.swift.compiler
(2 failures)

Actual outcome
The above mentioned error.

Expected outcome
When I run Carthage (carthage bootstrap --platform iOS --cache-builds --no-use-binaries) with the linker fix (orignally mentioned in #3097; not sure where that issue went), then the framework builds just fine.

The linker fix:

EXCLUDED_ARCHS__EFFECTIVE_PLATFORM_SUFFIX_simulator__NATIVE_ARCH_64_BIT_x86_64=arm64 arm64e armv7 armv7s armv6 armv8
EXCLUDED_ARCHS=$(inherited) $(EXCLUDED_ARCHS__EFFECTIVE_PLATFORM_SUFFIX_$(EFFECTIVE_PLATFORM_SUFFIX)__NATIVE_ARCH_64_BIT_$(NATIVE_ARCH_64_BIT))

I'm unsure if this is a bug in our framework or a bug in the latest changes.

The text was updated successfully, but these errors were encountered:

EmDee · 2021-02-04T23:07:04Z

What I forgot to mention:

If I move the correctly built AppAuth_iOS framework to the Carthage/Build/iOS directory, then I can suddenly build pace-cloud-sdk as well. I'm confused as to why only AppAuth is affected and not the other dependencies.

tovkal · 2021-02-06T19:15:51Z

I think this is because PACECloudSDK is referencing its dependencies as frameworks, which of course does not find in the Carthage/Build folder as all dependencies are compiled as XCFrameworks.

I am having the same issue with RxRealm. I am guessing this will be an issue until Carthage implements support for github binaries.

keithcml · 2021-02-07T16:39:20Z

I have got similar issue for frameworks referencing RxSwift. (like RxDataSources, RxOptional). Carthage cannot link the RxSwift.xcframework to RxDataSources .xcodeproj.

Missing linking to RxSwift.xcframework

scannillo · 2021-02-10T17:18:18Z

Hello. I am running into the same issue in developing the braintree-ios-drop-in SDK. It seems like we'll need to wait for Carthage to resolve this.

@EmDee If I move the correctly built AppAuth_iOS framework to the Carthage/Build/iOS directory, then I can suddenly build pace-cloud-sdk as well. I'm confused as to why only AppAuth is affected and not the other dependencies.

Can I ask - after you move the correctly built AppAuth_iOS framework into Carthage/Build/iOS, how are you then re-building the pace-cloud-sdk?

EmDee · 2021-02-10T19:10:48Z

Can I ask - after you move the correctly built AppAuth_iOS framework into Carthage/Build/iOS, how are you then re-building the pace-cloud-sdk?

@scannillo I just ran the Carthage command again: carthage bootstrap --platform iOS --cache-builds --no-use-binaries --use-xcframeworks.

But to be honest, we switched over to SPM as recommended integration.

zachwaugh · 2021-02-19T20:36:36Z

I've been hitting this as well with one of our private dependencies, and think I may have found the cause, at least for my case. The PR for XCFrameworks says:

When building, Carthage extracts platform-specific frameworks to a temporary directory and passes that directory to Xcode. This allows frameworks to link against nested dependencies without adding xcframeworks to their project

I built Carthage from source so I could debug, and looks like this not working in all cases, as it's not extracting the frameworks to a temp directory when I build my dependencies. This is where it's failing - https://github.com/Carthage/Carthage/blob/master/Source/CarthageKit/Xcode.swift#L842

guard let platformTripleOS = settings.platformTripleOS.value,
		    let frameworkSearchPaths = settings.frameworkSearchPaths.value,
		    frameworkSearchPaths.contains(where: isRelativeToBuildDirectory) else {
    // Skip extracting xcframeworks if this project doesn't declare its OS triple, or if it doesn't link
    // against any frameworks in Carthage/Build.
    return SignalProducer(value: nil)
}

Specifically the frameworkSearchPaths.contains(where: isRelativeToBuildDirectory) condition. With some logging, I see the urls and paths used within the check look like this (renamed and simplified slightly):

let isRelativeToBuildDirectory = { (url: URL) in
    url.resolvingSymlinksInPath().path.starts(with: buildDirectory.resolvingSymlinksInPath().path)
}

The url param here is the Framework Search Path:

URL: file:///Users/zachwaugh/project/Carthage/Checkouts/MyDependency/Carthage/Build/iOS/
Resolved Path: /Users/zachwaugh/project/Carthage/Checkouts/MyDependency/Carthage/Build/iOS

buildDirectory:

URL: file:///Users/zachwaugh/project/Carthage/Checkouts/MyDependency/Carthage/Build
Resolved Path: /Users/zachwaugh/project/Carthage/Build

So /Users/zachwaugh/project/Carthage/Checkouts/MyDependency/Carthage/Build/iOS does not start with /Users/zachwaugh/project/Carthage/Build and the check fails. If I remove that frameworkSearchPaths.contains(where: isRelativeToBuildDirectory) completely, everything works.

I'm assuming it fails because while /Users/zachwaugh/project/Carthage/Checkouts/MyDependency/Carthage/Build is a symlink to /Users/zachwaugh/project/Carthage/Build, there is no iOS directory present, so it fails when it tries to resolve symbolic link and just falls back to the original URL. So either a bug in that check, or I'm misunderstanding the intention there (or both). @elliottwilliams can you confirm if that is a bug or something else I'm doing wrong with how our dependencies are configured? My repo is private, but I can try to make a sample if it helps. I'm not sure the right fix, but guessing we need to check if the path ends with Carthage/Build/xxx and if so, remove the last path component first and then attempt to resolve the symlink or something along those lines.

YMMV, but a workaround in my case was to mkdir Carthage/Build/iOS before building so the symlink resolves correctly, and then everything worked correctly.

keithcml · 2021-02-23T10:02:42Z

I have some findings.

I am observing the DerivedData/project-xxx/Build/Products/Release-iphonesimulator,

Seems Carthage needs to copy corresponding arch .framework from xxx.xcframework/ to DerivedData/project-xxx/Build/Products/Release-iphonesimulator.

There should be some issues so that the project cannot be built as missing linking framework.

Maybe adding the scripts to copy to that directory works! So that no need to manually link with Xcode.

ddaddy · 2021-02-23T15:22:43Z

mkdir Carthage/Build/iOS fixed the issue for me, and also mkdir Carthage/Build/Mac for a Mac build.

elliottwilliams · 2021-02-24T07:40:36Z

I'm assuming it fails because while /Users/zachwaugh/project/Carthage/Checkouts/MyDependency/Carthage/Build is a symlink to /Users/zachwaugh/project/Carthage/Build, there is no iOS directory present, so it fails when it tries to resolve symbolic link and just falls back to the original URL. So either a bug in that check, or I'm misunderstanding the intention there (or both). @elliottwilliams can you confirm if that is a bug or something else I'm doing wrong with how our dependencies are configured?
[…]
YMMV, but a workaround in my case was to mkdir Carthage/Build/iOS before building so the symlink resolves correctly, and then everything worked correctly.

@zachwaugh thanks for investigating, I can reproduce this and can confirm your workaround!

At this line,

Carthage/Source/CarthageKit/Xcode.swift

Line 838 in 0668de4

    
           url.resolvingSymlinksInPath().path.starts(with: buildDirectory.resolvingSymlinksInPath().path)

we assume that resolvingSymlinksInPath() will resolve a path like Carthage/Build/iOS, even if it doesn't exist. I can't find documentation in Foundation about what happens when you try to resolve a nonexistent path, but the realpath(3) POSIX call says:

All components of file_name must exist when realpath() is called.

I would guess we're hitting those semantics, and will need to rethink this line.

So: this bug leads to build errors under --use-xcframeworks, and you can work around it by creating platform-specific directories, like: mkdir Carthage/Build/<platform>.

…irectory Projects may search a directory like Carthage/Build/iOS, but under --use-xcframeworks that platform-specific folder ("iOS") might not exist. resolvingSymlinksInPath() follows realpath(3) semantics and will not resolve symlinks in the components of a non-existant URL. Fixed in extractXCFrameworks() by truncating the given URL down to its extant portion. Since we only care about prefix matching, and since the Carthage/Build URL will exists at build time due to symlinkBuildPathIfNeeded, this is safe. Fixes Carthage#3122 and testing using the Cartfile in the issue.

lluisgerard · 2021-03-09T18:30:35Z

I have got similar issue for frameworks referencing RxSwift. (like RxDataSources, RxOptional). Carthage cannot link the RxSwift.xcframework to RxDataSources .xcodeproj.
Missing linking to RxSwift.xcframework

@keithcml Did you solve this issue with RxSwift? I also have RxDataSources (but it's failing first on RxGesture).
I checked my project and I do have Carthage/Build/iOS in my project and the symlink seems correct so I'm not sure what else to check. Thanks!

carthage bootstrap --platform iOS --cache-builds --use-xcframeworks --no-use-binaries

*** Building scheme "RxGesture-iOS" in RxGesture.xcodeproj
Build Failed
	Task failed with exit code 65:
...

log:

/Users/lluisgerard/Xcode/apps/notes-ios/Carthage/Checkouts/RxGesture/Pod/Classes/iOS/UITapGestureRecognizer+RxGesture.swift:22:8: error: no such module 'RxSwift'
import RxSwift
       ^

scannillo mentioned this issue Feb 10, 2021

Support building as XCFramework braintree/braintree-ios-drop-in#268

Closed

agirault mentioned this issue Feb 23, 2021

Auth0.swift does not compile with Carthage and Xcode 12 auth0/Auth0.swift#456

Closed

elliottwilliams changed the title ~~--use-xcframeworks doesn't resolve dependencies correctly~~ Build errors with nested dependencies and --use-xcframeworks Feb 24, 2021

elliottwilliams added bug possible-workaround labels Feb 24, 2021

elliottwilliams mentioned this issue Feb 24, 2021

Fix xcframework extraction when a project refers to a missing build directory #3135

Merged

elliottwilliams closed this as completed in #3135 Mar 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build errors with nested dependencies and --use-xcframeworks #3122

Build errors with nested dependencies and --use-xcframeworks #3122

EmDee commented Feb 4, 2021

EmDee commented Feb 4, 2021

tovkal commented Feb 6, 2021 •

edited

Loading

keithcml commented Feb 7, 2021

scannillo commented Feb 10, 2021

EmDee commented Feb 10, 2021

zachwaugh commented Feb 19, 2021

keithcml commented Feb 23, 2021 •

edited

Loading

ddaddy commented Feb 23, 2021 •

edited

Loading

elliottwilliams commented Feb 24, 2021

lluisgerard commented Mar 9, 2021

Build errors with nested dependencies and --use-xcframeworks #3122

Build errors with nested dependencies and --use-xcframeworks #3122

Comments

EmDee commented Feb 4, 2021

EmDee commented Feb 4, 2021

tovkal commented Feb 6, 2021 • edited Loading

keithcml commented Feb 7, 2021

scannillo commented Feb 10, 2021

EmDee commented Feb 10, 2021

zachwaugh commented Feb 19, 2021

keithcml commented Feb 23, 2021 • edited Loading

ddaddy commented Feb 23, 2021 • edited Loading

elliottwilliams commented Feb 24, 2021

lluisgerard commented Mar 9, 2021

tovkal commented Feb 6, 2021 •

edited

Loading

keithcml commented Feb 23, 2021 •

edited

Loading

ddaddy commented Feb 23, 2021 •

edited

Loading