Draft a Servicing Granularity overview doc

Documentation/planning/arcade-powered-source-build/README.md

@@ -42,6 +42,9 @@ build process is driven by two major goals:
      impact of changes, reducing the frequency the source-build servicing team
      has to root-cause and patch over problems.
 
+   * Gives us a place to add additional checks in the future, such as
+     [nongranular servicing readiness](../nongranular-servicing-readiness/).
+
 This doc is about where we can start, what incremental progress would look like,
 and the vision.
 

Documentation/planning/nongranular-servicing-readiness/README.md

@@ -0,0 +1,96 @@
+# Non-granular servicing readiness
+
+This doc describes the risks caused by servicing granularity differences between
+Microsoft and source-build, and proposes reducing the risk by adding tests to
+the Microsoft build ensuring source-build's lack of granularity doesn't cause
+problems. Alternative approaches are discussed in
+[ServicingGranularity-RejectedApproaches.md](ServicingGranularity-RejectedApproaches.md).
+
+This is related to [source-build#923 "Figure out how source-build will work with
+CoreFX per-package servicing
+policy"](https://github.com/dotnet/source-build/issues/923).
+
+## "Build only if changed" policy
+
+This is a maintenance strategy where if a *product* hasn't changed, it isn't
+rebuilt and won't be rereleased under a new patch version.
+
+> A *product* for the sake of this doc is any released asset where an
+independent decision is made whether that specific asset will be patched and
+shipped.
+
+For projects that release NuGet packages, it's ordinary to treat each NuGet
+package as a product. For example, when a new .NET Core SDK patch is being built
+to take a networking stack fix, the Microsoft servicing workflow doesn't involve
+building and releasing new MSBuild `.nupkg` files to the NuGet Gallery. When the
+patched SDK is built, old MSBuild binaries are retrieved from the NuGet Gallery.
+
+Linux distributions tend to follow the same strategy, but the product is the set
+of distro packages built from a single source package. To create a distro
+package release, the entire source package is built. If one of the patched
+distro packages takes a dependency built by a different source package, that
+source package doesn't need to be rebuilt: the old build of the dependency is
+used.
+
+The difference is that the Microsoft .NET Core SDK build is very granular: each
+NuGet package may be a product, even if it's built from the same source tree as
+other NuGet packages. This causes a conflict when Microsoft chooses to build and
+release a subset of what it considers the products contained in a specific
+source package, but Linux distro expectations are to release the entire product
+(all packages) from that source package. The partial build behavior is the
+default when building from a `vX.Y.Z` Git tag in a dotnet repository.
+
+## How source-build works now, despite different granularity
+
+The current approach is to disable the partial build behavior through a build
+flag while building the tag.
+
+🚩 This is a risky workaround with the current state of the repos. There's no
+mechanism in place that ensures the new packages will behave the same as the
+ones produced by earlier Microsoft releases. See [source-build#923 "Figure out
+how source-build will work with CoreFX per-package servicing
+policy"](https://github.com/dotnet/source-build/issues/923).
+
+## Can we unify servicing behavior?
+
+In general, we want to minimize differences between the Microsoft build and
+source-build. Making the builds behave the same way would be the natural way to
+resolve the issue.
+
+Applying the Microsoft build approach to source-build, we could split the
+source-build .NET Core SDK into many source packages (and in turn, many distro
+packages). This is considered unmaintainable. (See ['Switching to microsoft
+build
+granularity'](ServicingGranularity-RejectedApproaches.md#switching-to-microsoft-build-granularity).)
+
+The other direction, making the Microsoft build use source-build's less granular
+policy, is likely a non-starter. It would force unnecessary redownload of
+products with no meaningful changes, a regression affecting products such as
+Visual Studio.
+
+Neither direction appears to be viable.
+
+## Reduce the risk and continue disabling partial build behavior
+
+We can continue with how things are now, but add tests to validate the different
+approaches don't cause incompatibilities.
+
+We should run the usual tests against the source-build outputs, to know the SDK
+and runtimes function as expected. This is a base level of validation.
+
+However, that would miss compatibility issues between source-build and the
+Microsoft build. For example, if source-build produces a 3.0.2 version of some
+assembly, but Microsoft hasn't released a patch to that assembly since 3.0.0, it
+may be possible for a dev to publish a project that depends on the 3.0.2 binary
+and fails to run on the Microsoft .NET Core Runtime because it only has 3.0.0.
+
+Additional testing may be able to cover compatibility problems. If the .NET Core
+repos know the version Microsoft last shipped of each product (already the case
+in some repos), tests can download the last Microsoft-shipped packages and
+compare the contents to the current. Tests can also analyze source code diffs to
+spot suspicious unreleased changes.
+
+Reversing the perspective may help clarify. Microsoft has an atypical servicing
+policy where some source packages are not fully built. The proposal is to add
+tests that help ensure compatibility when Microsoft chooses not to build part of
+a source package.

Documentation/planning/nongranular-servicing-readiness/ServicingGranularity-RejectedApproaches.md

@@ -0,0 +1,130 @@
+# Servicing Granularity - Rejected Approaches
+See [README.md](README.md) for context on the servicing granularity problem.
+This document discusses rejected approaches.
+
+# Switching to microsoft build granularity
+This section discusses the consequences of switching to the very granular
+Microsoft servicing policy.
+
+## Background: granularity of the Microsoft servicing build
+How granular is the Microsoft build really? Each dotnet repo is generally a
+product, but some repositories go further, containing multiple products in a
+single source tree. This is an example based on 3.0 (some products may be
+missing from the list):
+
+* CoreFX
+  * The NETCoreApp shared framework implementation (product)
+    * `runtime.linux-x64.Microsoft.Private.CoreFx.NETCoreApp`
+    * Always built when NETCoreApp gets any patch.
+  * The out of band (OOB) packages (*each* is a product)
+    * `System.Data.SqlClient`
+    * `System.Diagnostics.EventLog`
+    * `System.Drawing.Common`
+    * `System.IO.Pipelines`
+    * `System.IO.Ports`
+    * ...more. These `System.*` packages are produced in CoreFX, but aren't
+      included in the NETCoreApp framework. Some are used by downstream
+      frameworks like ASP.NET Core, and by the tools and SDK. They are also used
+      by apps directly via `PackageReference`.
+    * These are somewhat likely to get patches over time.
+
+* Core-Setup
+  * The NETCoreApp shared framework and Runtime/AppHost pack (product)
+    * `Microsoft.NETCore.App.Runtime.linux-x64`
+    * Always built when NETCoreApp gets any patch.
+  * The NETCoreApp Targeting pack (product)
+  * NETStandard Targeting pack (product)
+    * Targeting packs are unlikely to get patches.
+
+* AspNetCore
+  * ASP.NET Core shared framework and Runtime pack (product)
+    * `Microsoft.AspNetCore.App.Runtime.linux-x64`
+  * ASP.NET Core Targeting pack (product)
+
+This means if you assemble a dependency graph of a theoretical .NET Core SDK,
+say 3.0.103, it will contain multiple versions of the same repos:
+
+* .NET Core SDK `v3.0.103`
+  * CoreFX
+    * `v3.0.3` - System.Security.Permissions, shared framework implementation
+    * `v3.0.2` - System.ServiceProcess.ServiceController
+    * `v3.0.1` - System.Security.Cryptography.Xml
+    * `v3.0.0` - System.IO.Pipelines
+  * [...]
+
+This shows how the number of nodes will naturally grow over time. The max number
+of versions of a specific repo is bounded by the number of products it builds.
+The max is only three for Core-Setup, but larger (dozens?) for CoreFX because of
+the number of independent OOB packages.
+
+## Consequences of depending on multiple versions of the same repo
+
+### Microsoft builds
+This is completely fine for Microsoft servicing builds, because they use
+prebuilts. When building a repo, all upstream products are downloaded from an
+online source, so the *version* of each product isn't important.
+
+### Building from source
+Multiple versions of the same repo has significant consequences for
+source-build, on the other hand. Downloading prebuilt products is forbidden, so
+every upstream product must be built from source. The more commits built, the
+longer the build takes.
+
+In some cases, we can redistribute the previous source-built binaries, but we
+have to account for bootstrapping on new distros (no previous source-build
+exists) and distros that don't keep old binaries around:
+
+> Some (not all) Linux distribution do keep around older distro packages in
+their distro repository (eg, we build source-build 3.0.10, and 3.0.9 is still
+around for installation). If that was a general policy (it's not, for example,
+in Fedora) than source-build could do what Microsoft build does and use older
+builds. Aside from being not being supported everywhere, it also means
+source-build needs previous patch versions to have been built in that distro
+too. A new distro couldn't skip packaging 3.0.0 if 3.0.10 still needs something
+from there. [[omajid's review
+comment](https://github.com/dotnet/source-build/pull/1389#discussion_r350783942)]
+
+Even worse, the number of dependency nodes will tend to increase over time as a
+branch gets serviced, as demonstrated above with the list of tags comprising a
+theoretical `v3.0.103` SDK. This means the difficulty and complexity of building
+.NET Core would then depend on how many times the branch has had a servicing
+release.
+
+A few concrete reasons this causes significant overhead and/or upfront work for
+source-build infra maintainers:
+
+1. Say a new distro appears that can't build anything earlier than CoreFX
+   `v3.0.3`, but we need to bootstrap .NET Core. To build the required `v3.0.0`,
+   `v3.0.1`, and `v3.0.2` tags, we must add patches to each one to get them to
+   build. The patching work *probably* only needs to be done once and applied on
+   all affected commits, but as the release branch continues to be serviced, the
+   amount of work to get the product built on a new distro compounds.
+2. Each repo must be checked out at every required commit/tag, which syncs a
+   *lot* of irrelevant source code.
+3. The repo must either be built in its entirety, or we need significant infra
+   work to allow source-build to pick the exact product it needs to build.
+
+## Q&A
+
+### Can we split up source-build products to make this less difficult?
+The thought here is that we might be able to split source-build into more
+tarballs with more intermediate distro packages. This could reduce the amount of
+code being compiled at once, making the build more incremental.
+
+There is direct feedback from experienced distro maintainers that we need to
+keep it simple: continue building a ".NET Core SDK + Runtime" product. Asking
+distro maintainers to keep track of many products is a significant burden and
+expected to cause mistakes.
+
+This also doesn't solve (1) from above. We would still need to add patches to
+old, unmaintained tags to bring up .NET Core on a new distro.
+
+### Does source-build-reference-packages help?
+Context: any NuGet package that only contains references can be built by
+[dotnet/source-build-reference-packages](https://github.com/dotnet/source-build-reference-packages)
+rather than checking out and building the original repo. In some cases, such as
+targeting packs, we already do this.
+
+This doesn't work for CoreFX OOB packages, which contain implementation. The OOB
+packages are a particular pain point with this approach, so
+source-build-reference-packages doesn't help significantly.