RFC for multi test run finalization

RFCs/0031-Test-Run-Attachments-Processing.md

@@ -0,0 +1,209 @@
+# 0031 Test Run Attachments Processing
+
+# Summary
+This document details a data collector extensibility point to reprocess (combine/merge) attachments obtained across test executions in both design mode and commandline scenarios.
+
+# Motivation
+Today when Test Platform executes tests in parallel only code coverage reports are merged (data collector attachments with uri: `datacollector://microsoft/CodeCoverage/2.0`). For other data collector attachments reprocessing is skipped and all of them are returned by Test Platform.
+
+The [dotnet test](https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test) command is used to execute unit tests in a given solution. The `dotnet test` command builds the solution and runs a test host application for each test project in the solution. However, currently there is no way to reprocess(combine/merge) data collector attachments associated with each project execution. Code coverage reports are not merged.
+
+When `Run All Tests` is performed in VS, tests for projects can be executed separately based on target platform/target framework amongst other criteria. In this case also combining/merging of data collector attachments is not performed (e.g. code overage reports are not merged). `Analyze Code Coverage for All Tests` is showing coverage report for only some test projects in the run.
+
+# Proposed Changes
+
+Introduce a new `IDataCollectorAttachmentProcessor` interface which can be implemented by Test Platform extensions and provide custom logic to reprocess(combine/merge) data collector attachments. Test Platform will invoke `ProcessAttachmentSetsAsync` only if at least 1 data collector attachment related to processor (through `GetExtensionUris`) is created by test execution.
+
+```
+namespace Microsoft.VisualStudio.TestPlatform.ObjectModel.DataCollection
+{
+    /// <summary>
+    /// Interface for data collectors add-ins that choose to reprocess generated attachments
+    /// </summary>
+    public interface IDataCollectorAttachmentProcessor
+    {
+        /// <summary>
+        /// Gets the attachments Uris, which are handled by attachment processor
+        /// </summary>
+        IEnumerable<Uri> GetExtensionUris();
+
+        /// <summary>
+        /// Indicates whether attachment processor is supporting incremental processing of attachments
+        /// </summary>
+        bool SupportsIncrementalProcessing   { get; }
+
+        /// <summary>
+        /// Reprocess attachments generated by independent test executions
+        /// </summary>
+        /// <param name="attachments">Attachments to be processed</param>
+        /// <param name="progressReporter">Progress reporter. Accepts integers from 0 to 100</param>
+        /// <param name="logger">Message logger</param>
+        /// <param name="cancellationToken">Cancellation token</param>
+        /// <returns>Attachments after reprocessing</returns>
+        Task<ICollection<AttachmentSet>> ProcessAttachmentSetsAsync(ICollection<AttachmentSet> attachments, IProgress<int> progressReporter, IMessageLogger logger, CancellationToken cancellationToken);
+    }
+}
+```
+
+Method `GetExtensionUris` should provide all Uris for data collector attachments which are handled by current attachment processor. Test platform will provide to attachment processor only data collector attachments with such Uris. Result of method `ProcessAttachmentSetsAsync` should contain only data collector attachments with such Uris.
+
+`SupportsIncrementalProcessing` should indicate if attachment processor is supporting incremental processing of attachments. It means that `ProcessAttachmentSetsAsync` should be [associative](https://en.wikipedia.org/wiki/Associative_property).
+
+If `SupportsIncrementalProcessing` is `True` Test Platform may try to speed up whole process by reprocessing data collector attachments as soon as possible when any two test executions are done. For example let's assume we have 5 test executions which are generating 5 data collector attachments: `a1`, `a2`, `a3`, `a4` and `a5`. Test platform could perform invocations:
+* `var result1 = await ProcessAttachmentSetsAsync([a1, a2, a3], ...);` when first 3 executions are done
+* `var result2 = await ProcessAttachmentSetsAsync(result1.Concat([a4]), ...);` when 4th execution is done
+* `var finalResult = await ProcessAttachmentSetsAsync(result2.Concat([a5]), ...);` when last test execution is done
+
+If `SupportsIncrementalProcessing` is `False` then Test Platform will wait for all test executions to finish and call `ProcessAttachmentSetsAsync` only once:
+* `var finalResult = await ProcessAttachmentSetsAsync([a1, a2, a3, a4, a5], ...);`
+
+By default `SupportsIncrementalProcessing` should be `False`, unless processing can take longer time and it's beneficial to start the process as soon as possible.
+
+
+
+2. Introduce a new `ProcessTestRunAttachmentsAsync` method in [IVsTestConsoleWrapper](https://github.com/microsoft/vstest/blob/master/src/Microsoft.TestPlatform.VsTestConsole.TranslationLayer/Interfaces/IVsTestConsoleWrapper.cs) interface:
+
+```
+/// <summary>
+/// Provides back all attachments to TestPlatform for additional processing (for example merging)
+/// </summary>
+/// <param name="attachments">Collection of attachments</param>
+/// <param name="processingSettings">XML processing settings</param>
+/// <param name="isLastBatch">Indicates that all test executions are done and all data is provided</param>
+/// <param name="collectMetrics">Enables metrics collection (used for telemetry)</param>
+/// <param name="eventsHandler">EventHandler to receive session complete event</param>
+/// <param name="cancellationToken">Cancellation token</param>        
+Task ProcessTestRunAttachmentsAsync(IEnumerable<AttachmentSet> attachments, string processingSettings, bool isLastBatch, bool collectMetrics, ITestRunAttachmentsProcessingEventsHandler eventsHandler, CancellationToken cancellationToken);
+```
+
+Method can be used to start a new test run attachments processing, which is reprocessing all data collector attachments passed as first argument using all available attachment processors. When `collectMetrics` is set to `true` Test Platform will provide information about initial number of attachments, final number of attachments, time taken in seconds to process all data collector attachments.
+
+3. Introduce a new `ITestRunAttachmentsProcessingEventsHandler` interface:
+```
+namespace Microsoft.VisualStudio.TestPlatform.ObjectModel.Client
+{
+    /// <summary>
+    /// Interface contract for handling test run attachments processing events
+    /// </summary>
+    public interface ITestRunAttachmentsProcessingEventsHandler : ITestMessageEventHandler
+    {
+        /// <summary>
+        /// Dispatch TestRunAttachmentsProcessingComplete event to listeners.
+        /// </summary>
+        /// <param name="attachmentsProcessingCompleteEventArgs">AttachmentsProcessing Complete event args.</param>
+        /// <param name="attachments">Last set of processed attachment sets.</param>
+        void HandleTestRunAttachmentsProcessingComplete(TestRunAttachmentsProcessingCompleteEventArgs attachmentsProcessingCompleteEventArgs, IEnumerable<AttachmentSet> lastChunk);
+
+        /// <summary>
+        /// Dispatch ProcessedAttachmentsChunk event to listeners.
+        /// </summary>
+        /// <param name="attachments">Processed attachment sets.</param>
+        void HandleProcessedAttachmentsChunk(IEnumerable<AttachmentSet> attachments);
+
+        /// <summary>
+        /// Dispatch TestRunAttachmentsProcessingProgress event to listeners.
+        /// </summary>
+        /// <param name="AttachmentsProcessingProgressEventArgs">AttachmentsProcessing Progress event args.</param>
+        void HandleTestRunAttachmentsProcessingProgress(TestRunAttachmentsProcessingProgressEventArgs AttachmentsProcessingProgressEventArgs);
+    }
+}
+```
+Interface provides callbacks from test run attachments processing. For every such process `HandleTestRunAttachmentsProcessingComplete` will be called once and will provide last chunk or all data collector attachments. During attachments processing `HandleProcessedAttachmentsChunk` can be invoked several times providing data collector attachments that are already processed. Method `HandleTestRunAttachmentsProcessingProgress` will be invoked every time when `progressReporter` is used by attachment processor and will provide information about current attachment processor: progress, uris and index of it. Additionally event will contain also number of attachment processors.
+
+
+4. Use above logic to reprocess data collector attachments for parallel test executions and VS scenarios (e.g. `Run All Tests`, `Analyze Code Coverage for All Tests`). In case of `Analyze Code Coverage for All Tests` VS will use `vstest.console` in a variation of design mode and merge all code coverage reports. VS will show full code coverage report for all test projects.
+
+5. When [dotnet test](https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test) command is used to execute unit tests in a given solution, attachments will also be reprocessed. Details regarding this process will be provided in separate RFC.
+
+# Additional classes
+
+1. `TestRunAttachmentsProcessingCompleteEventArgs` used by ITestRunAttachmentsProcessingEventsHandler:
+
+```
+namespace Microsoft.VisualStudio.TestPlatform.ObjectModel.Client
+{
+    [DataContract]
+    public class TestRunAttachmentsProcessingCompleteEventArgs : EventArgs
+    {
+        /// <summary>
+        /// Default constructor.
+        /// </summary>
+        /// <param name="isCanceled">Specifies whether the attachments processing is canceled.</param>
+        /// <param name="error">Specifies the error encountered during the execution of the attachments processing.</param>
+        public TestRunAttachmentsProcessingCompleteEventArgs(bool isCanceled, Exception error)
+        {
+            this.IsCanceled = isCanceled;
+            this.Error = error;
+        }
+
+        /// <summary>
+        /// Gets a value indicating whether the attachments processing is canceled or not.
+        /// </summary>
+        [DataMember]
+        public bool IsCanceled { get; private set; }
+
+        /// <summary>
+        /// Gets the error encountered during the attachments processing of the test runs. Null if there is no error.
+        /// </summary>
+        [DataMember]
+        public Exception Error { get; private set; }
+
+        /// <summary>
+        /// Get or Sets the Metrics (used for telemetry)
+        /// </summary>
+        [DataMember]
+        public IDictionary<string, object> Metrics { get; set; }
+    }
+}
+```
+
+2. `TestRunAttachmentsProcessingProgressEventArgs` used by ITestRunAttachmentsProcessingEventsHandler:
+
+```
+namespace Microsoft.VisualStudio.TestPlatform.ObjectModel.Client
+{
+    [DataContract]
+    public class TestRunAttachmentsProcessingProgressEventArgs : EventArgs
+    {
+        /// <summary>
+        /// Default constructor.
+        /// </summary>
+        /// <param name="currentAttachmentProcessorIndex">Specifies current attachment processor index.</param>
+        /// <param name="currentAttachmentProcessorUris">Specifies current processor Uris.</param>
+        /// <param name="currentAttachmentProcessorProgress">Specifies current processor progress.</param>
+        /// <param name="attachmentProcessorsCount">Specifies the overall number of processors.</param>
+        public TestRunAttachmentsProcessingProgressEventArgs(long currentAttachmentProcessorIndex, ICollection<Uri> currentAttachmentProcessorUris, long currentAttachmentProcessorProgress, long attachmentProcessorsCount)
+        {
+            CurrentAttachmentProcessorIndex = currentAttachmentProcessorIndex;
+            CurrentAttachmentProcessorUris = currentAttachmentProcessorUris;
+            CurrentAttachmentProcessorProgress = currentAttachmentProcessorProgress;
+            AttachmentProcessorsCount = attachmentProcessorsCount;
+        }
+
+        /// <summary>
+        /// Gets a current attachment processor index.
+        /// </summary>
+        [DataMember]
+        public long CurrentAttachmentProcessorIndex { get; private set; }
+
+        /// <summary>
+        /// Gets a current attachment processor URI.
+        /// </summary>
+        [DataMember]
+        public ICollection<Uri> CurrentAttachmentProcessorUris { get; private set; }
+
+        /// <summary>
+        /// Gets a current attachment processor progress.
+        /// </summary>
+        [DataMember]
+        public long CurrentAttachmentProcessorProgress { get; private set; }
+
+        /// <summary>
+        /// Gets the overall number of attachment processors.
+        /// </summary>
+        [DataMember]
+        public long AttachmentProcessorsCount { get; private set; }
+    }
+}
+
+```