[VMR] Add cross build support #3671

steveisok · 2023-10-20T02:09:25Z

Today, the VMR is set up to build for linux-x64. In order to build all supported .NET configurations within the VMR, we need to have the ability to target os/architectures that are different from the build host.

T-Shirt Size: Large

omajid · 2023-10-26T17:02:09Z

FYI @Swapnali911

directhex · 2023-10-26T17:03:29Z

I got this working today, with two patches on top of those mentioned in previous issues, plus passing the --use-mono-runtime flag to source-build.

directhex · 2023-10-27T15:09:54Z

OK, so, status update:

The full patch I'm applying looks like this:

diff --git a/src/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj b/src/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
index 7122c81034..924b1e7d45 100644
--- a/src/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
+++ b/src/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj
@@ -11,7 +11,6 @@
   <PropertyGroup Condition="'$(BuildingInsideVisualStudio)' != 'true'">
     <PublishDir>$(RuntimeBinDir)ilc-published/</PublishDir>
     <!-- Can't use NativeAOT in source build yet https://github.com/dotnet/runtime/issues/66859 -->
-    <NativeAotSupported Condition="'$(DotNetBuildFromSource)' == 'true'">false</NativeAotSupported>
     <!-- Disable native AOT on FreeBSD when cross building from Linux. -->
     <NativeAotSupported Condition="'$(TargetOS)' == 'freebsd' and '$(CrossBuild)' == 'true'">false</NativeAotSupported>
     <PublishAot Condition="'$(NativeAotSupported)' == 'true'">true</PublishAot>
diff --git a/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj b/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
index ef1b82f537..9789cefda2 100644
--- a/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
+++ b/src/runtime/src/coreclr/tools/aot/crossgen2/crossgen2.csproj
@@ -5,11 +5,9 @@
   <PropertyGroup>
     <OutputPath>$(RuntimeBinDir)crossgen2</OutputPath>
     <!-- Can't use NativeAOT in source build yet https://github.com/dotnet/runtime/issues/66859 -->
-    <NativeAotSupported Condition="'$(DotNetBuildFromSource)' == 'true'">false</NativeAotSupported>
     <!-- Trimming is not currently working, but set the appropriate feature flags for NativeAOT -->
     <PublishTrimmed Condition="'$(NativeAotSupported)' == 'true'">true</PublishTrimmed>
-    <RuntimeIdentifiers Condition="'$(_IsPublishing)' != 'true' and '$(DotNetBuildFromSource)' != 'true'">linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;freebsd-arm64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64</RuntimeIdentifiers>
-    <RuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">$(PackageRID)</RuntimeIdentifiers>
+    <RuntimeIdentifiers Condition="'$(_IsPublishing)' != 'true'">linux-x64;linux-musl-x64;linux-arm;linux-musl-arm;linux-arm64;linux-musl-arm64;freebsd-x64;freebsd-arm64;osx-x64;osx-arm64;win-x64;win-x86;win-arm64</RuntimeIdentifiers>
     <SelfContained>false</SelfContained>
     <SelfContained Condition="'$(_IsPublishing)' == 'true'">true</SelfContained>
   </PropertyGroup>
diff --git a/src/runtime/eng/native/build-commons.sh b/src/runtime/eng/native/build-commons.sh
index d9eb04cd8a..3dcae0f254 100755
--- a/src/runtime/eng/native/build-commons.sh
+++ b/src/runtime/eng/native/build-commons.sh
@@ -514,6 +514,7 @@ while :; do
 
     shift
 done
+__PortableBuild=1
 
 if [[ -z "$__HostArch" ]]; then
     __HostArch=$__TargetArch
diff --git a/src/installer/src/redist/targets/GenerateLayout.targets b/src/installer/src/redist/targets/GenerateLayout.targets
index 9ce7496291..bfe353f1b0 100644
--- a/src/installer/src/redist/targets/GenerateLayout.targets
+++ b/src/installer/src/redist/targets/GenerateLayout.targets
@@ -38,7 +38,7 @@
 
       <CoreSetupRid Condition="'$(CoreSetupRid)' == ''">$(HostRid)</CoreSetupRid>
       <CoreSetupRid Condition=" ('$(OSName)' == 'win' or '$(OSName)' == 'osx' or '$(OSName)' == 'freebsd') and '$(DotNetBuildFromSource)' != 'true' ">$(OSName)-$(Architecture)</CoreSetupRid>
-      <CoreSetupRid Condition="$(HostRid.StartsWith('mariner.2.0'))">$(HostRid.Replace('mariner.2.0', 'cm.2'))</CoreSetupRid>
+      <!-- wtf <CoreSetupRid Condition="$(HostRid.StartsWith('mariner.2.0'))">$(HostRid.Replace('mariner.2.0', 'cm.2'))</CoreSetupRid>-->
 
       <!-- only the runtime OSX .pkgs have a `-internal` suffix -->
       <InstallerStartSuffix Condition="$([MSBuild]::IsOSPlatform('OSX'))">-internal</InstallerStartSuffix>
diff --git a/repo-projects/runtime.proj b/repo-projects/runtime.proj
index 67959e981d..cfe10b9dc7 100644
--- a/repo-projects/runtime.proj
+++ b/repo-projects/runtime.proj
@@ -22,10 +22,11 @@
     <BaseOS>$(NETCoreSdkPortableRuntimeIdentifier.Substring(0, $(_platformIndex)))</BaseOS>
 
     <BuildCommandArgs>$(StandardSourceBuildArgs)</BuildCommandArgs>
+    <BuildCommandArgs>$(BuildCommandArgs) --arch arm64</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:TargetRid=$(OverrideTargetRid)</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:RuntimeOS=$(RuntimeOS)</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:BaseOS=$(BaseOS)</BuildCommandArgs>
-    <BuildCommandArgs>$(BuildCommandArgs) /p:SourceBuildNonPortable=true</BuildCommandArgs>
+    <BuildCommandArgs>$(BuildCommandArgs) /p:SourceBuildNonPortable=false</BuildCommandArgs>
     <BuildCommandArgs>$(BuildCommandArgs) /p:UsingToolMicrosoftNetCompilers=false</BuildCommandArgs>
     <BuildCommand>$(StandardSourceBuildCommand) $(BuildCommandArgs)</BuildCommand>
   </PropertyGroup>

The csproj changes are needed for restore to work on runtime. --arch being hardcoded in runtime.proj is obviously wrong, but it needs adding somehow. The other changes are from one of my other issues.

I'm building with docker run -e CROSSCOMPILE=1 -e ROOTFS_DIR=/crossrootfs/arm64 -w pwd--platform linux/amd64 -vpwd:pwd -it mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64 ./build.sh --online --use-mono-runtime -- /p:OverrideTargetRid=linux-arm64

There is obviously a bunch of architecture confusion still in the build:

artifacts in artifacts/x64/Release/
some of those artifacts have -x64 filename suffixes
Private.SourceBuilt.Prebuilts.8.0.100-rtm.23526.1.mariner.2.0-x64.tar.gz contains an x64 runtime pack, as well as arm64
Some files in artifacts/obj/x64/Release/blob-feed/assets/ have x64 suffix, especially symbols packages

Also, the build only works with Mono runtime, because one of the things the CoreCLR build tries to build is a component called crosscomponents, for x64 (build arch). We need to figure out what to do for this if we want CoreCLR to work.

But... it seems functional for a proof-of-concept. I also tested the patch above on Mac, and got an osx-arm64 build when using arch -x86_64 to force x64 host platform detection

steveisok · 2023-10-30T20:56:11Z

@directhex do you have any suggestions for the top level switches? OverrideTargetRid, that works, but it seems verbose to me.

directhex · 2023-10-30T20:58:07Z

Probably generate the OverrideTargetRid from either /p:TargetOS and /p:TargetArchitecture values, or move it even higher level and add --os and --arch to build.sh

steveisok · 2023-11-02T14:05:37Z

OK, so, status update:

The full patch I'm applying looks like this:

@directhex what do you think of PR'ing this patch and put it under the DotNetBuildVertical flag?

directhex · 2023-11-02T14:07:38Z

Is that the flag we decided on? I saw there was ongoing discussion on Teams

steveisok · 2023-11-02T14:15:11Z

We'll talk a little more just to make sure everyone agrees. I can't see it changing significantly.

steveisok · 2023-11-02T16:22:14Z

Is that the flag we decided on? I saw there was ongoing discussion on Teams

Since there was no pushback after our sync, let's roll with DotNetBuildVertical. Your patch would be the best place to introduce it, I think.

Swapnali911 · 2023-11-03T12:27:14Z

Hi I have tried cross build for ppc64le , but currently failing

Commencing build of "  iltools " target in "CoreCLR component" for linux.ppc64le.Release in /home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.ppc64le.Release
      Invoking "/home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/eng/native/gen-buildsys.sh" "/home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr" "/home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.ppc64le.Release" ppc64le linux clang Release ""  -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="linux" -DFEATURE_DISTRO_AGNOSTIC_SSL=1  -DCLR_CMAKE_KEEP_NATIVE_SYMBOLS=true
      ~/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.ppc64le.Release ~/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr
      Not searching for unused variables given on the command line.
      loading initial cache file /home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/eng/native/tryrun.cmake
      -- The C compiler identification is unknown
      -- The CXX compiler identification is unknown
      CMake Error at CMakeLists.txt:9 (project):
        The CMAKE_C_COMPILER:
 
          /usr/local/bin/clang-16
 
        is not a full path to an existing compiler tool.
 
        Tell CMake where to find the compiler by setting either the environment
        variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
        the compiler, or to the compiler name if it is in the PATH.
 
 
      CMake Error at CMakeLists.txt:9 (project):
        The CMAKE_CXX_COMPILER:
 
          /usr/local/bin/clang++
 
        is not a full path to an existing compiler tool.
 
        Tell CMake where to find the compiler by setting either the environment
        variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
        to the compiler, or to the compiler name if it is in the PATH.
 
 
      -- Configuring incomplete, errors occurred!
      ~/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr
      ~/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.ppc64le.Release ~/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr
      Executing make   iltools  -j 8
      make: *** No rule to make target 'iltools'.  Stop.
      ~/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr
      ##vso[task.logissue type=error]Failed to build "CoreCLR component".
    /home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/runtime.proj(94,5): error MSB3073: The command ""/home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/build-runtime.sh" -ppc64le -release -ci -portablebuild=false -keepnativesymbols -os linux -outputrid linux-ppc64le -component iltools" exited with code 2.
 
    Build FAILED.
 
    /home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/runtime.proj(94,5): error MSB3073: The command ""/home/ubuntu/swapnali/check_net8/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/build-runtime.sh" -ppc64le -release -ci -portablebuild=false -keepnativesymbols -os linux -outputrid linux-ppc64le -component iltools" exited with code 2.
        0 Warning(s)
        1 Error(s)
 
    Time Elapsed 00:00:51.78
    Build failed with exit code 1. Check errors above

Patch : 3671.patch

Steps

Clone dotnet/dotnet repo
Checkout to rc2 tag
Apply patch for ppc64le arch
prep.sh
export CROSSCOMPILE=1
export ROOTFS_DIR=/crossrootfs/ppc64le
./build.sh --use-mono-runtime --online -- /p:OverrideTargetRid=linux-ppc64le

Swapnali911 · 2023-11-16T16:21:24Z

Above error resolved for runtime after clang version is upgraded to clang-16, runtime build is successful
Build is currently failing for aspnetcore

[14:43:33.25] Building 'aspnetcore'
Running command:
/home/ubuntu/swapnali/check_net8/dotnet/src/aspnetcore/eng/build.sh --ci --configuration Release --restore --build --pack -bl /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath="/home/ubuntu/swapnali/check_net8/dotnet/artifacts/obj/x64/Release/PackageVersions.aspnetcore.props" /p:AdditionalSourceBuiltNupkgCacheDir="/home/ubuntu/swapnali/check_net8/dotnet/artifacts/obj/x64/Release/blob-feed/packages/" /p:ReferencePackageNupkgCacheDir="/home/ubuntu/swapnali/check_net8/dotnet/prereqs/packages/reference/" /p:PreviouslySourceBuiltNupkgCacheDir="/home/ubuntu/swapnali/check_net8/dotnet/prereqs/packages/previously-source-built/" /p:SourceBuildUseMonoRuntime=true --arch ppc64le --no-build-repo-tasks --no-build-nodejs /p:PublishCompressedFilesPathPrefix=/home/ubuntu/swapnali/check_net8/dotnet/artifacts/obj/x64/Release/blobs/aspnetcore/Runtime/ /p:PortableBuild=true /p:TargetRuntimeIdentifier=linux-ppc64le /p:MicrosoftNetFrameworkReferenceAssembliesVersion=1.0.0

Error :
/home/ubuntu/swapnali/check_net8/dotnet/src/aspnetcore/artifacts/source-build/self/src/src/Tools/dotnet-user-jwts/src/dotnet-user-jwts.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 8.0.0-rc.2.23475.17) [/home/ubuntu/swapnali/check_net8/dotnet/.dotnet/sdk/8.0.100-rc.1.23455.8/NuGet.targets]

./artifacts/obj/x64/Release/blob-feed/packages/Microsoft.NETCore.App.Host.linux-ppc64le.8.0.0-rc.2.23475.17.nupkg is available but ASP.NET is looking for Microsoft.NETCore.App.Host.linux-x64

mmitche · 2024-01-17T22:30:25Z

@directhex Is this work largely complete? Could this be closed out? What's left?

directhex · 2024-01-18T00:15:49Z

IMHO the broader "add support" issue is done. Any lingering bugs with the cross build (e.g. incorrect artifact or folder names used during the build) should be filed and fixed as distinct issues rather than keeping this open.

Swapnali911 · 2024-01-18T04:55:56Z

HelloWorld program run is failing on ppc64le with tarball generated by cross building dotnet/dotnet repo
#3797 : Can someone look into this. Thanks.

tkapin · 2024-01-29T15:55:10Z

@directhex - since the last update is 2 weeks ago, could you please share the status? Are we on track to get this finished by 14 Feb? Cheers!

directhex · 2024-01-29T15:56:45Z

I think this can be closed. We support cross-building as a general concept, and have it working in CI.

steveisok added the area-unified-build label Oct 20, 2023

steveisok added this to .NET Unified Build Oct 20, 2023

github-project-automation bot added this to .NET Source Build Oct 20, 2023

github-project-automation bot moved this to Backlog in .NET Source Build Oct 20, 2023

dotnet-issue-labeler bot added the untriaged label Oct 20, 2023

steveisok assigned directhex Oct 20, 2023

steveisok removed this from .NET Source Build Oct 20, 2023

steveisok removed the untriaged label Oct 20, 2023

mmitche mentioned this issue Oct 27, 2023

VMR output and intermediate directory names need a rework #3696

Closed

steveisok moved this to In Progress in .NET Unified Build Oct 30, 2023

steveisok mentioned this issue Oct 30, 2023

[VMR] linux/mac-arm64 cross build fails w/ CoreCLR runtime #3698

Closed

steveisok mentioned this issue Oct 30, 2023

[VMR] Linux Vertical Build PoC #3699

Closed

9 tasks

This comment was marked as duplicate.

Sign in to view

directhex mentioned this issue Nov 6, 2023

Build portable runtime when PortableBuild is forced in SB/VMR dotnet/installer#17698

Merged

tmds mentioned this issue Nov 20, 2023

Installer for v8.0.100 fails to restore/build: Missing Microsoft.NET.Sdk.Aspire.Manifest-8.0.100 dotnet/installer#17797

Closed

directhex mentioned this issue Nov 29, 2023

Add Mariner cross-compile VMR builds dotnet/installer#17891

Merged

Swapnali911 mentioned this issue Dec 15, 2023

[VMR] Issue with cross built tarball for Ppc64le #3797

Closed

github-project-automation bot added this to .NET Source Build Jan 3, 2024

github-project-automation bot moved this to Backlog in .NET Source Build Jan 3, 2024

tkapin mentioned this issue Jan 11, 2024

.NET Unified Build (Theme) #3926

Open

MichaelSimons removed this from .NET Source Build Jan 9, 2024

tkapin added the Epic Groups multiple user stories. Can be grouped under a theme. label Jan 15, 2024

mmitche closed this as completed Jan 29, 2024

github-project-automation bot moved this from In Progress to Done in .NET Unified Build Jan 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VMR] Add cross build support #3671

[VMR] Add cross build support #3671

steveisok commented Oct 20, 2023 •

edited by mmitche

Loading

omajid commented Oct 26, 2023

directhex commented Oct 26, 2023

directhex commented Oct 27, 2023

steveisok commented Oct 30, 2023

directhex commented Oct 30, 2023

steveisok commented Nov 2, 2023

directhex commented Nov 2, 2023

steveisok commented Nov 2, 2023

This comment was marked as duplicate.

steveisok commented Nov 2, 2023

Swapnali911 commented Nov 3, 2023

Swapnali911 commented Nov 16, 2023

mmitche commented Jan 17, 2024

directhex commented Jan 18, 2024 via email

Swapnali911 commented Jan 18, 2024

tkapin commented Jan 29, 2024

directhex commented Jan 29, 2024

[VMR] Add cross build support #3671

[VMR] Add cross build support #3671

Comments

steveisok commented Oct 20, 2023 • edited by mmitche Loading

omajid commented Oct 26, 2023

directhex commented Oct 26, 2023

directhex commented Oct 27, 2023

steveisok commented Oct 30, 2023

directhex commented Oct 30, 2023

steveisok commented Nov 2, 2023

directhex commented Nov 2, 2023

steveisok commented Nov 2, 2023

This comment was marked as duplicate.

steveisok commented Nov 2, 2023

Swapnali911 commented Nov 3, 2023

Swapnali911 commented Nov 16, 2023

mmitche commented Jan 17, 2024

directhex commented Jan 18, 2024 via email

Swapnali911 commented Jan 18, 2024

tkapin commented Jan 29, 2024

directhex commented Jan 29, 2024

steveisok commented Oct 20, 2023 •

edited by mmitche

Loading