Use live apphost when publishing ilc as singlefile

[am11]

Using the following command to cross-build product for FreeBSD:

docker run --rm -v$(pwd)/runtime:/runtime -e ROOTFS_DIR=/crossrootfs/x64 \
  mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-amd64-freebsd-14 \
  sh -c '/runtime/build.sh clr+libs+packs -c Debug -os freebsd -cross &&
  tdnf install -y file unzip && cd /tmp &&
  unzip /runtime/artifacts/packages/Debug/Shipping/runtime.freebsd-x64.Microsoft.DotNet.ILCompiler.9.0.0-dev.nupkg &&
  find . -name ilc -exec file {} \;'

before:

./tools/ilc: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=771fdbe539f221e5b1e4e6dc000846f10d503b3f, stripped

after:

./tools/ilc: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400097), FreeBSD-style, BuildID[sha1]=4b265d82eb3e1de5315251f7a5db2713ced94d2e, stripped

---

It required defer building ILCompiler.csproj until after apphost sfxproj. Also, PackageRID is used to restore the "host" package to build the target, while OutputRID is used to create package for the target, which is what we need here.

Fix #104497

[jkotas]

The tricky ordering of the different build steps looks fragile.

I am wondering whether it would be better to use multiple build.sh invocations to bootstrap the FreeBSD cross build. The first invocation would skip building parts that depend on last-known-good packages for target arch, the second invocation would then build the rest and use the packages produced by the first invocation.

[am11]

Yup, it is fragile as in, when we are not building packs subset, it fails the build on platforms using PublishSingleFile. Pulling Microsoft.NETCore.App.Host.sfxproj in clr subset is also not an option because that depends on libs+host to build first.

There is an existing bug that on the systems where we use PublishAot for ILCompiler, it still looks for apphost (#80154 (comment), it was actually a regression in .net9 p1). That makes the refactoring those parts a degree more delicate.

[am11]

@jkotas, the ILCompiler.csproj build will be now deferred until after packs are built, only for freebsd/illumos/haiku OS set which do not provide Microsoft SDK. This requires single invocation of ./build.sh.

The current state of this PR (351856c) is such that if we still don't like Subsets.props changes, then we can revert Subsets.props (alone) and document the following steps for the devs of those platforms:

# invocation 1 - build everything without ILCompiler support
$ ./build.sh -os freebsd -p:NativeAotSupported=false

# invocation 2 - build clr.tools and packs subsets again without NativeAotSupported
#                condition so ILCompiler.csproj can use live-built apphost.
$ ./build.sh clr.tools+packs -os freebsd

[jkotas]

Ideally, the cross-arch and cross-OS bootstrapping builds would produce a full-featured bits that are equivalent (including how the tools get published) to the natively compiled packages. I think achieving it via multiple invocations of build.sh flow would be easier to maintain than the ordering of the individual build steps.

[am11]

@Thefrank, @sec, basically this is the new workflow for community platforms for cross-OS build (e.g. for freebsd on linux): #105004 (comment) (two steps). I tested it with this workflow https://github.com/am11/CrossRepoCITesting/actions/runs/10845732989/workflow, and artifacts are https://github.com/am11/CrossRepoCITesting/releases/tag/freebsd_10845732989. The ilc executable in runtime.freebsd-x64.Microsoft.DotNet.ILCompiler.10.0.0-ci.nupkg shows:

$ unzip ~/Downloads/runtime.freebsd-x64.Microsoft.DotNet.ILCompiler.10.0.0-ci.nupkg
$ find . -name ilc -exec file {} \; | fold
./tools/ilc: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamic
ally linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400097), FreeB
SD-style, BuildID[sha1]=235f8e6220cbf08ec8c1966a415f82f6ed072e4e, with debug_inf
o, not stripped

[jkotas]

$ ./build.sh -os freebsd -p:NativeAotSupported=false
$ ./build.sh clr.tools+packs -os freebsd

I think we need to have a clean separation of responsibilities between the two commands. Otherwise, I expect it will collide with where #107772 is trying to go. A good split may be:

• The first command should be partial build that produces packages that the full build expects to come from LKG toolsets when there is one for target platform
• The second command should be a full build that uses the packages produced by the first command

We can define a new subset for the first command to make the command line UX simpler and stable over time. This subset would not be on by default. Building this subset can drop the packages in a new directory. The full build can automatically pick the LKG toolset from there if the directory exists.

Ideally, it would be even possible for the first command to be debug/checked build and the second command to be release build or vice versa, and it should work for non-cross builds too.

@agocke Thoughts?

[am11]

@jkotas, @jkoritzinsky, community platform builds are green. Note that freebsd-x64 and linux-loongarch64 set NativeAotSupported as before. If #110688 is merged before this PR, then we can add targetArchitecture==riscv64 condition in Subsets.props and enable it. Otherwise, I will add it in that PR or as a separate one once both are merged.

If we want to enable UseNativeAotForComponents in the CI, then we will need to add a third stage; first stage prepares apphost, second stage uses PublishSingleFile for tools build and the third stage will use tools from second stage to use PublishAot. Please let me know if we want the third stage.

[jkotas]

Is this a problem for incremental builds? Should the build with LKG target and the build local targeting pack go into separate directories?

[am11]

This project is not built in stage 1, so incremental build does not come into play.

[jkotas]

I am not following why we would need 3 stages to enable UseNativeAotForComponents. The tools built by the first stage need to target the build machine architecture only to unblock second stage. We should have everything available to build the tools that target the build machine architecture during the first stage.

[am11]

I am not following why we would need 3 stages to enable UseNativeAotForComponents. The tools built by the first stage need to target the build machine architecture only to unblock second stage. We should have everything available to build the tools that target the build machine architecture during the first stage.

@jkotas, I have tested it. Setting UseNativeAotForComponents alone did not use live AOT build as expected, it just published crossgen2/ilc with corehost. Seems like it requires more configurations like we have in tests.singlefile.targets etc. so AOT toolchain can find PrivateSdkAssemblies and FrameworkAssemblies.

[am11]

@jkoritzinsky ILCompiler_publish.csproj is not listed in this file (or anywhere else). Was it an oversight or something for the future?

[jkoritzinsky]

I think there was an intent to hook it up for the Microsoft.DotNet.ILCompiler package, but I think it didn't quite happen. Looks like we still publish from the ILCompiler.csproj project and ship that. I think the build needs to be adjusted to hook this up correctly (use ILCompiler_publish.csproj to publish ilc and remove the publishing logic from ILCompiler.csproj)

[jkotas]

Seems like it requires more configurations like we have in tests.singlefile.targets etc. so AOT toolchain can find PrivateSdkAssemblies and FrameworkAssemblies.

This should work by building the NAOT compiler and runtime packs. We may want to force PublishAotUsingRuntimePack to true when these packages are used in the second stage to side-step the problem of building the compiler for the target arch in the first stage.

Replacing PrivateSdkAssemblies and FrameworkAssemblies requires hooking up the rest of the build integration too that often has changes that has to be in sync with the runtime. It gets complicated and fragile quickly. We used to do that, but got rid of it.

[am11]

PublishAotUsingRuntimePack

It gives

/runtime/.dotnet/sdk/10.0.100-alpha.1.24610.7/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(497,5):
error NETSDK1082: There was no runtime pack for Microsoft.NETCore.App available for the specified RuntimeIdentifier 'linux-loongarch64'.
[/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]

We can look into UseNativeAotForComponents separately. Original issue #104497 was about corehost published ilc binary is using linux apphost on freebsd cross build, which is fixed.

[am11]

I think we should invert this condition; blacklist instead of whitelist the unsupported platforms.