[System.Runtime.Loader] Add hot reload test infrastructure (#51144)

Adding infrastructure for hot reload testing.

For each test we define a new library assembly project.  The `.csproj` has a `DeltaScript` property that specifies a JSON file that lists the name of an initial source file, and a list of updated versions of that file.  The https://github.com/dotnet/hotreload-utils Microsoft.DotNet.HotReload.Utils.Generator.BuildTool nuget .targets file will run a generator tool during the build to read the delta script and creates metadata, IL and PDB deltas that incorporate the updates.

The main testsuite references all the test assemblies, and when a test runs, it calls `ApplyUpdateUtil.ApplyUpdate` to load subsequent deltas and then compares the results before and after an update.

Dependencies:

- https://github.com/dotnet/hotreload-utils  - there is now a `Microsoft.DotNet.HotReload.Utils.Generator.BuildTool` MSBuild targets nupkg on the `dotnet6-transport` nuget feed.
- If `DOTNET_MODIFIABLE_ASSEMBLIES` is not set in the environment, we use remote executor to run the tests.

Mono is technically enabled, but practically it's not running anywhere:
- We have not enabled the MonoMetadataUpdate feature support on desktop linux or macos
- For wasm, ios and Android since we don't have remote executor support, we would need to adjust the test pipelines to pass the `DOTNET_MODIFIABLE_ASSEMBLIES` flag via xharness (and the app builder tasks).
- For ios and Android we would need to enable the interpreter as the execution engine, otherwise the tests are skipped.

To try it out locally with CoreCLR run:

```
./dotnet.sh build src/libraries/System.Runtime.Loader/tests /t:Test
```

For mono, build the runtime with `/p:MonoMetadataUpdate=true` and setenv `MONO_ENV_OPTIONS=--interp`

---


* [System.Runtime.Loader] Add hot reload test infrastructure

   Adding infrastructure for hot reload testing.

* Make a proper task for computing hotreload-delta-gen output files

* Don't need DeltaCount property, compute from json

   Compute the number of deltas that hotreload-delta-gen will produce by parsing the json script and counting the number of changes.

* Add dependency on hotreload-delta-gen tool

* use 'dotnet tool run hotreload-delta-gen' to generate EnC deltas

* Use remote executor if DOTNET_MODIFIABLE_ASSEMBLIES is not set

* Use DotNetTool property to run hotreload-delta-gen

   just `dotnet tool run hotreload-delta-gen` can fail on 6.0 preview 2 because of borked argument parsing

* Don't run on Mono for now

   1. mobile devices and wasm will need xharness or the app builder tasks to set the DOTNET_MODIFIABLE_ASSEMBLIES environment variable.
   2. for console apps, we need the hot reload capabilities API to check if the current runtime can do updates

* bump hotreload-delta-gen package version

* Run tests on Mono if feature enabled and interp is used

* fix remote executor detection

   just calling RemoteExecutor.IsSupported on wasm throws

   ```
   System.TypeInitializationException: The type initializer for 'Microsoft.DotNet.RemoteExecutor.RemoteExecutor' threw 
an exception.
   ---> System.PlatformNotSupportedException: System.Diagnostics.Process is not supported on this platform.
   ```

* remove unneeded DefineConstants

* Use `$(TargetPath)` as input to ComputeDeltaOutputNames

   Co-authored-by: Viktor Hofer <[email protected]>

* drop "BuildingProject" property check

   It's for the old project style, only

* Use Microsoft.DotNet.HotReload.Utils.Generator.BuildTool instead of hotreload-delta-gen

   Use an msbuild target nuget instead of the CLI tool

* Use well known version for Generator BuiltTool version

* Use published Generator.BuildTool package

* fix typos, indentantion; add copyright headers

* Remove ComputeDeltaFileOutputNames use nuget version

   The Microsoft.DotNet.HotReload.Utils.Generator.BuildTool nuget now includes a task to compute the output names.  So the targets in the nuget are entirely self-contained now - incremental builds and project references should work now.

   Bump to version 1.0.1 of Generator.BuildTool which has the necessary targets

* Fix style nits; add more copyright headers

Co-authored-by: Viktor Hofer <[email protected]>

eng/Version.Details.xml

@@ -210,5 +210,9 @@
       <Uri>https://github.com/dotnet/emsdk</Uri>
       <Sha>e51b2a920817aec200d7306b95f616f9451d47a3</Sha>
     </Dependency>
+    <Dependency Name="Microsoft.DotNet.HotReload.Utils.Generator.BuildTool" Version="1.0.1-alpha.0.21257.1">
+      <Uri>https://github.com/dotnet/hotreload-utils</Uri>
+      <Sha>078f7f63b604e97c64cf907e9932a85f8aeb5dc3</Sha>
+    </Dependency>
   </ToolsetDependencies>
 </Dependencies>

eng/Versions.props

@@ -149,6 +149,7 @@
     <MicrosoftNETTestSdkVersion>16.9.0-preview-20201201-01</MicrosoftNETTestSdkVersion>
     <MicrosoftDotNetXHarnessTestRunnersXunitVersion>1.0.0-prerelease.21255.1</MicrosoftDotNetXHarnessTestRunnersXunitVersion>
     <MicrosoftDotNetXHarnessCLIVersion>1.0.0-prerelease.21255.1</MicrosoftDotNetXHarnessCLIVersion>
+    <MicrosoftDotNetHotReloadUtilsGeneratorBuildToolVersion>1.0.1-alpha.0.21257.1</MicrosoftDotNetHotReloadUtilsGeneratorBuildToolVersion>
     <XUnitVersion>2.4.1</XUnitVersion>
     <XUnitRunnerVisualStudioVersion>2.4.2</XUnitRunnerVisualStudioVersion>
     <CoverletCollectorVersion>1.3.0</CoverletCollectorVersion>

src/libraries/System.Runtime.Loader/tests/ApplyUpdate/Directory.Build.props

@@ -0,0 +1,15 @@
+<Project>
+  <Import Project="..\..\Directory.Build.props" />
+
+  <ItemGroup>
+    <!-- This package from https://github.com/dotnet/hotreload-utils provides
+	 targets that read the json delta script and generates deltas based on the baseline assembly and the modified sources.
+	 
+	 Projects must define the DeltaScript property that specifies the (relative) path to the json script.
+	 Deltas will be emitted next to the output assembly.  Deltas will be copied when the current
+	 project is referenced from other other projects.
+    --> 
+    <PackageReference Include="Microsoft.DotNet.HotReload.Utils.Generator.BuildTool" Version="$(MicrosoftDotNetHotReloadUtilsGeneratorBuildToolVersion)" />
+  </ItemGroup>
+
+</Project>

src/libraries/System.Runtime.Loader/tests/ApplyUpdate/Directory.Build.targets

@@ -0,0 +1,3 @@
+<Project>
+  <Import Project="..\..\..\Directory.Build.targets" />
+</Project>

src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1/MethodBody1.cs

@@ -0,0 +1,11 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+namespace System.Reflection.Metadata.ApplyUpdate.Test
+{
+    public class MethodBody1 {
+        public static string StaticMethod1 () {
+            return "OLD STRING";
+        }
+    }
+}

src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1/MethodBody1_v1.cs

@@ -0,0 +1,11 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+namespace System.Reflection.Metadata.ApplyUpdate.Test
+{
+    public class MethodBody1 {
+        public static string StaticMethod1 () {
+            return "NEW STRING";
+        }
+    }
+}

src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1/MethodBody1_v2.cs

@@ -0,0 +1,11 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+namespace System.Reflection.Metadata.ApplyUpdate.Test
+{
+    public class MethodBody1 {
+        public static string StaticMethod1 () {
+            return "NEWEST STRING";
+        }
+    }
+}

src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1/System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1.csproj

@@ -0,0 +1,11 @@
+<Project Sdk="Microsoft.NET.Sdk">
+  <PropertyGroup>
+    <RootNamespace>System.Runtime.Loader.Tests</RootNamespace>
+    <TargetFrameworks>$(NetCoreAppCurrent)</TargetFrameworks>
+    <TestRuntime>true</TestRuntime>
+    <DeltaScript>deltascript.json</DeltaScript>
+  </PropertyGroup>
+  <ItemGroup>
+    <Compile Include="MethodBody1.cs" />
+  </ItemGroup>
+</Project>

src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1/deltascript.json

@@ -0,0 +1,7 @@
+{
+    "changes": [
+        {"document": "MethodBody1.cs", "update": "MethodBody1_v1.cs"},
+        {"document": "MethodBody1.cs", "update": "MethodBody1_v2.cs"},
+    ]
+}
+

src/libraries/System.Runtime.Loader/tests/ApplyUpdateTest.cs

@@ -0,0 +1,42 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using Xunit;
+
+namespace System.Reflection.Metadata
+{
+    ///
+    /// The general setup for ApplyUpdate tests is:
+    ///
+    /// Each test Foo has a corresponding assembly under
+    /// System.Reflection.Metadata.ApplyUpate.Test.Foo The Foo.csproj has a delta
+    /// script that applies one or more updates to Foo.dll The ApplyUpdateTest
+    /// testsuite runs each test in sequence, loading the corresponding
+    /// assembly, applying an update to it and observing the results.
+    [Collection(nameof(ApplyUpdateUtil.NoParallelTests))]
+    [ConditionalClass(typeof(ApplyUpdateUtil), nameof (ApplyUpdateUtil.IsSupported))]
+    public class ApplyUpdateTest
+    {
+        [Fact]
+        void StaticMethodBodyUpdate()
+        {
+            ApplyUpdateUtil.TestCase(static () =>
+            {
+                var assm = typeof (ApplyUpdate.Test.MethodBody1).Assembly;
+
+                var r = ApplyUpdate.Test.MethodBody1.StaticMethod1();
+                Assert.Equal("OLD STRING", r);
+
+                ApplyUpdateUtil.ApplyUpdate(assm);
+
+                r = ApplyUpdate.Test.MethodBody1.StaticMethod1();
+                Assert.Equal("NEW STRING", r);
+
+                ApplyUpdateUtil.ApplyUpdate(assm);
+
+                r = ApplyUpdate.Test.MethodBody1.StaticMethod1 ();
+                Assert.Equal ("NEWEST STRING", r);
+            });
+        }
+    }
+}

src/libraries/System.Runtime.Loader/tests/ApplyUpdateUtil.cs

@@ -0,0 +1,148 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+using Xunit;
+using Microsoft.DotNet.RemoteExecutor;
+
+namespace System.Reflection.Metadata
+{
+    public class ApplyUpdateUtil {
+        internal const string DotNetModifiableAssembliesSwitch = "DOTNET_MODIFIABLE_ASSEMBLIES";
+        internal const string DotNetModifiableAssembliesValue = "debug";
+
+        [CollectionDefinition("NoParallelTests", DisableParallelization = true)]
+        public class NoParallelTests { }
+
+        /// Whether ApplyUpdate is supported by the environment, test configuration, and runtime.
+        ///
+        /// We need:
+        /// 1. Either DOTNET_MODIFIABLE_ASSEMBLIES=debug is set, or we can use the RemoteExecutor to run a child process with that environment; and,
+        /// 2. Either Mono in a supported configuration (interpreter as the execution engine, and the hot reload feature enabled), or CoreCLR; and,
+        /// 3. The test assemblies are compiled in the Debug configuration.
+        public static bool IsSupported => (IsModifiableAssembliesSet || IsRemoteExecutorSupported) &&
+            (!IsMonoRuntime || IsSupportedMonoConfiguration) &&
+            IsSupportedTestConfiguration();
+
+        public static bool IsModifiableAssembliesSet =>
+            String.Equals(DotNetModifiableAssembliesValue, Environment.GetEnvironmentVariable(DotNetModifiableAssembliesSwitch), StringComparison.InvariantCultureIgnoreCase);
+
+        // static cctor for RemoteExecutor throws on wasm.
+        public static bool IsRemoteExecutorSupported => !RuntimeInformation.IsOSPlatform(OSPlatform.Create("BROWSER")) && RemoteExecutor.IsSupported;
+
+        // copied from https://github.com/dotnet/arcade/blob/6cc4c1e9e23d5e65e88a8a57216b3d91e9b3d8db/src/Microsoft.DotNet.XUnitExtensions/src/DiscovererHelpers.cs#L16-L17
+        private static readonly Lazy<bool> s_isMonoRuntime = new Lazy<bool>(() => Type.GetType("Mono.RuntimeStructs") != null);
+        public static bool IsMonoRuntime => s_isMonoRuntime.Value;
+
+        private static readonly Lazy<bool> s_isSupportedMonoConfiguration = new Lazy<bool>(CheckSupportedMonoConfiguration);
+
+        public static bool IsSupportedMonoConfiguration => s_isSupportedMonoConfiguration.Value;
+
+        // Not every build of Mono supports ApplyUpdate
+        internal static bool CheckSupportedMonoConfiguration()
+        {
+        // check that interpreter is enabled, and the build has hot reload capabilities enabled.
+            var isInterp = RuntimeFeature.IsDynamicCodeSupported && !RuntimeFeature.IsDynamicCodeCompiled;
+            return isInterp && HasApplyUpdateCapabilities();
+        }
+
+        internal static bool HasApplyUpdateCapabilities()
+        {
+            var ty = typeof(AssemblyExtensions);
+            var mi = ty.GetMethod("GetApplyUpdateCapabilities", BindingFlags.NonPublic | BindingFlags.Static, Array.Empty<Type>());
+
+            if (mi == null)
+                return false;
+
+            var caps = mi.Invoke(null, null);
+
+            // any non-empty string, assumed to be at least "baseline"
+            return caps is string {Length: > 0};
+        }
+
+        // Only Debug assemblies are editable
+        internal static bool IsSupportedTestConfiguration()
+        {
+#if DEBUG
+            return true;
+#else
+            return false;
+#endif
+        }
+
+        private static System.Collections.Generic.Dictionary<Assembly, int> assembly_count = new();
+
+        internal static void ApplyUpdate (System.Reflection.Assembly assm)
+        {
+            int count;
+            if (!assembly_count.TryGetValue(assm, out count))
+                count = 1;
+            else
+                count++;
+            assembly_count [assm] = count;
+
+            /* FIXME WASM: Location is empty on wasm. Make up a name based on Name */
+            string basename = assm.Location;
+            if (basename == "")
+                basename = assm.GetName().Name + ".dll";
+            Console.Error.WriteLine($"Apply Delta Update for {basename}, revision {count}");
+
+            string dmeta_name = $"{basename}.{count}.dmeta";
+            string dil_name = $"{basename}.{count}.dil";
+            byte[] dmeta_data = System.IO.File.ReadAllBytes(dmeta_name);
+            byte[] dil_data = System.IO.File.ReadAllBytes(dil_name);
+            byte[] dpdb_data = null; // TODO also use the dpdb data
+
+            AssemblyExtensions.ApplyUpdate(assm, dmeta_data, dil_data, dpdb_data);
+        }
+
+        internal static bool UseRemoteExecutor => !IsModifiableAssembliesSet;
+
+        internal static void AddRemoteInvokeOptions (ref RemoteInvokeOptions options)
+        {
+                options = options ?? new RemoteInvokeOptions();
+                options.StartInfo.EnvironmentVariables.Add(DotNetModifiableAssembliesSwitch, DotNetModifiableAssembliesValue);
+        }
+
+        /// Run the given test case, which applies updates to the given assembly.
+        ///
+        /// Note that the testBody should be a static delegate or a static
+        /// lambda - it must not use state from the enclosing method.
+        public static void TestCase(Action testBody,
+                                    RemoteInvokeOptions options = null)
+        {
+            if (UseRemoteExecutor)
+            {
+                Console.Error.WriteLine ($"Running test using RemoteExecutor");
+                AddRemoteInvokeOptions(ref options);
+                RemoteExecutor.Invoke(testBody, options).Dispose();
+            }
+            else
+            {
+                Console.Error.WriteLine($"Running test using direct invoke");
+                testBody();
+            }
+        }
+
+        /// Run the given test case, which applies updates to the given
+        /// assembly, and has 1 additional argument.
+        ///
+        /// Note that the testBody should be a static delegate or a static
+        /// lambda - it must not use state from the enclosing method.
+        public static void TestCase(Action<string> testBody,
+                                    string arg1,
+                                    RemoteInvokeOptions options = null)
+        {
+            if (UseRemoteExecutor)
+            {
+                AddRemoteInvokeOptions(ref options);
+                RemoteExecutor.Invoke(testBody, arg1, options).Dispose();
+            }
+            else
+            {
+                testBody(arg1);
+            }
+        }
+    }
+}

src/libraries/System.Runtime.Loader/tests/System.Runtime.Loader.Tests.csproj

@@ -8,6 +8,8 @@
     <GenerateDependencyFile>false</GenerateDependencyFile>
   </PropertyGroup>
   <ItemGroup>
+    <Compile Include="ApplyUpdateTest.cs" />
+    <Compile Include="ApplyUpdateUtil.cs" />
     <Compile Include="AssemblyExtensionsTest.cs" />
     <Compile Include="AssemblyLoadContextTest.cs" />
     <Compile Include="CollectibleAssemblyLoadContextTest.cs" />
@@ -34,6 +36,7 @@
     <ProjectReference Include="ReferencedClassLibNeutralIsSatellite\ReferencedClassLibNeutralIsSatellite.csproj" />
     <ProjectReference Include="LoaderLinkTest.Shared\LoaderLinkTest.Shared.csproj" />
     <ProjectReference Include="LoaderLinkTest.Dynamic\LoaderLinkTest.Dynamic.csproj" />
+    <ProjectReference Include="ApplyUpdate\System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1\System.Reflection.Metadata.ApplyUpdate.Test.MethodBody1.csproj" />
   </ItemGroup>
   <ItemGroup Condition="'$(TargetOS)' == 'Browser'">
     <WasmFilesToIncludeFromPublishDir Include="$(AssemblyName).dll" />