CoreCLR packages are not servicing ready

[ericstj]

Recently identified in #111444

All packages under https://github.com/dotnet/runtime/tree/7bb850f5f660e815aefe5a0937c6c29635820af8/src/coreclr/.nuget are not set up to ship in servicing.

If they're meant to be enabled/disabled incrementally then a process around versioning and enabling needs to be documented and followed for servicing changes.

If they're meant to always ship, then changes to the build need to be made to enable them and consumers need to be set up to react to those regular changes.

[dotnet-policy-service]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

[ericstj]

This may apply for packages other than those in coreclr/.nuget. Those are just the ones we noticed. It makes sense to do an audit of all packages maintained by runtime to see if they have a process for servicing. That way you can discover gaps before the day they need to be serviced.

[hoyosjs]

@ericstj - some of them are meant to be published every time. What stopped producing them/marking them as shipping?

[ericstj]

They've always been that way if you look at the publishing history, for example https://www.nuget.org/packages/runtime.win-x64.Microsoft.NETCore.ILDAsm

I suspect if you want them to build with every release you might need to determine if that would have any side-effects - make sure the packages are setup to flow / version correctly / etc.

[JulieLeeMSFT]

@amanasifkhalid, PTAL.
CC @jkoritzinsky.

[amanasifkhalid]

some of them are meant to be published every time. What stopped producing them/marking them as shipping?

Are there any that shouldn't be shipped for servicing?

I suspect if you want them to build with every release you might need to determine if that would have any side-effects - make sure the packages are setup to flow / version correctly / etc.

Aside from kicking off an official build, is there anything else we can/should do to verify these packages can be shipped before the next servicing release?

[jkotas]

Are there any that shouldn't be shipped for servicing?

The ones that ship as part of GA should be shipped as servicing too.

Number of packages in this directory are non shipping, for example, https://github.com/dotnet/runtime/blob/main/src/coreclr/.nuget/ILCompiler.Reflection.ReadyToRun.Experimental/ILCompiler.Reflection.ReadyToRun.Experimental.pkgproj#L5 . This package does not ship as part of GA and it should not be shipped during serving either.

[amanasifkhalid]

The ones that ship as part of GA should be shipped as servicing too.

Thanks -- I suppose we should start by removing the catch-all rule here, and let the package configs handle themselves. Would getting a successful official build with this change on top of a copy of the release/9.0-staging branch be a good starting point for verifying this change? Or do servicing builds use a different process?

[JulieLeeMSFT]

We think there is no different process for servicing builds.
CC @carlossanlop.

[amanasifkhalid]

I ran an official build of release/9.0-staging with the blanket rule removed from src/coreclr/.nuget/Directory.Build.props, and I can verify the same CoreCLR packages available in GA are under PackageArtifacts.

Number of packages in this directory are non shipping, for example, https://github.com/dotnet/runtime/blob/main/src/coreclr/.nuget/ILCompiler.Reflection.ReadyToRun.Experimental/ILCompiler.Reflection.ReadyToRun.Experimental.pkgproj#L5 . This package does not ship as part of GA and it should not be shipped during serving either.

I'm finding this package available under PackageArtifacts in every official non-servicing build (example) -- I'm guessing this shouldn't be the case?

[ViktorHofer]

The PackageArtifacts folder contains both the non-shipping and shipping packages. The non-shipping ones go to a transport feed. That's intentional. You should find a MergedManifest.xml under BlobArtifacts which should tell you which packages are marked as shipping.

[amanasifkhalid]

Thanks @ViktorHofer! The shipping statuses in the MergedManifest.xml from my official build look good:

<Package Id="dotnet-pgo" Version="9.0.3-servicing.25126.13" DotNetReleaseShipping="false" NonShipping="true"/>
<Package Id="ILCompiler.Reflection.ReadyToRun.Experimental" Version="9.0.3-servicing.25126.13" DotNetReleaseShipping="false" NonShipping="true"/>
...
<Package Id="Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>

Here's what the manifest looks like for an earlier servicing build:

<Package Id="dotnet-pgo" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>
<Package Id="ILCompiler.Reflection.ReadyToRun.Experimental" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>
...
<Package Id="Microsoft.NETCore.ILAsm" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>
<Package Id="Microsoft.NETCore.ILDAsm" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>

I'm guessing we want the version suffix for ILAsm/ILDasm set as well. I'll fix that.

[amanasifkhalid]

I'm guessing we want the version suffix for ILAsm/ILDasm set as well. I'll fix that.

After messing with this locally, I'm guessing the desired package version is passed during the build (i.e. /p:PackageVersion=x.y.z). Is it sufficient to just allow the StableVersion of shipped servicing packages to be set here in src\coreclr\.nuget\Directory.Build.targets\? With that approach, the version suffix for shipped packages is 9.0.3 in my local builds. For an actual servicing release, would we manually specify the package version anyway?

[ericstj]

I did an audit of all the packages that ship in ga and which were missing from servicing and found the complete list here: https://gist.github.com/ericstj/2be44b58fab7026875164a5e66020910

Have a review in case that includes any other packages owned by area-Infrastructure-coreclr

For an actual servicing release, would we manually specify the package version anyway?

That really depends on what you want the behavior to be. If you want them to always ship and follow the product versioning (preferred) then you shouldn't specify any versions manually.
If you instead wanted to selectively enable and disable, then you'd want to specify the version independently.

At one point long ago in coreclr branch these packages built using the latter policy. I think you can stop doing that since folks forgot how to do it anyways. Ship all the time. Really the only downside of doing that is if your downstream consumers understand the regular updates and they work well for them. If those consumers are part of the product for instance - make sure they haven't done something like hardcode Major.0.0.

[amanasifkhalid]

Thanks @ericstj for putting together a list! I think we have all the CoreCLR packages covered. From the build manifest:

<Package Id="dotnet-ilverify" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.ILVerification" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NET.Sdk.IL" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x86.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x86.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>

For that particular build, the IsStable property is false -- is that why these packages all lack version suffixes? I suspect for stable builds, the correct suffix will be appended, such as in this build.

[ViktorHofer]

For that particular build, the IsStable property is false -- is that why these packages all lack version suffixes? I suspect for stable builds, the correct suffix will be appended.

That's the expectation, yes. We should double check when we do the next servicing release that these have indeed the correct version suffix. Or we queue a test stable build - unsure if that's worth the hassle.

---

We should follow-up on the remaining non-runtime packages (sourcelink, vs-codecoverage and llvm-project). Ignore test-templates as that repo got merged into sdk and is unlikely to change in the active servicing branches.

For sourcelink, I don't think anything in the repo infrastructure prevents the packages from getting built during servicing. This might be a case where the release infra team intentionally doesn't flow updates during servicing. cc @mmitche @leecow