Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] In assembly parallel #296

Merged
merged 33 commits into from
Nov 29, 2017
Merged
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
41c9616
WIP: In assembly parallelization at method level.
Oct 20, 2017
621da1b
WIP: Make ClassInitialize/Cleanup and AssemblyInitialize/Cleanup sync…
Oct 20, 2017
87c41cb
Run test class/assembly cleanups at end of execution.
Oct 20, 2017
912cc54
Test Class parallelization.
AbhitejJohn Oct 20, 2017
95e912a
Added E2E tests.
AbhitejJohn Oct 20, 2017
9377a41
Merge pull request #2 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Oct 20, 2017
550b70d
Runsettings support.
AbhitejJohn Oct 20, 2017
560411e
Made Parallel platform independent.
AbhitejJohn Oct 21, 2017
6262793
Merge pull request #3 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Oct 21, 2017
69a2392
nit refactoring and test fixes.
AbhitejJohn Oct 21, 2017
9bd125b
Another nit fix.
AbhitejJohn Oct 21, 2017
c442547
Fixing test script to fail on test failures.
AbhitejJohn Oct 21, 2017
93f3d8b
Some cleanup and a few UTs.
AbhitejJohn Oct 22, 2017
26d68c8
Merge pull request #4 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Oct 22, 2017
fa60189
Meged latest master.
AbhitejJohn Oct 30, 2017
4f1dcef
few PR comments
AbhitejJohn Oct 30, 2017
8c5b283
Merge pull request #5 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 10, 2017
ec72649
Changes based on spec.
AbhitejJohn Nov 20, 2017
053969f
Exception handling for settings, unit tests and Disable Parallelization.
AbhitejJohn Nov 20, 2017
4b37cba
Merge pull request #6 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 20, 2017
5b51599
Fixed up E2E tests.
AbhitejJohn Nov 21, 2017
7555ed1
Adding TestExecutionManager unit tests.
AbhitejJohn Nov 21, 2017
7821679
Merge pull request #7 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 21, 2017
ee36c06
Reverting the UWP sdk upgrade since PR build machines are not on the …
AbhitejJohn Nov 21, 2017
59882e6
Merge pull request #8 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 21, 2017
a2f5bfc
Updating localized files.
AbhitejJohn Nov 21, 2017
1b9a78b
Merge pull request #9 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 21, 2017
fcecc7c
Staging changes for STA/MTA of parallized tests and a test.
AbhitejJohn Nov 21, 2017
d26387a
The TaskScheduler.FromCurrentSyncContext did not work. It appears tha…
AbhitejJohn Nov 24, 2017
575ccc1
Merge pull request #10 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 24, 2017
4a4637e
Merge branch 'master' into in-assembly-parallel
smadala Nov 27, 2017
2f9c7ba
Really feel bad adding a sleep to a test - This can probably be handl…
AbhitejJohn Nov 28, 2017
256151e
Merge pull request #11 from AbhitejJohn/in-assembly-parallel
AbhitejJohn Nov 28, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Made Parallel platform independent.
Support for DoNotParallelize at assembly level.
Fixed up a loc'ing issue with test assembly being loaded in the main app domain.
Runsettings parallel level taking precedence over assembly level setting.
AbhitejJohn committed Oct 21, 2017

Verified

This commit was signed with the committer’s verified signature. The key has expired.
aramase Anish Ramasekar
commit 560411e4cc5e5577022bf3fda8e08d5aa0a1eedc
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution
{
using System;
using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers;
using Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel;

internal class TestAssemblySettingsProvider : MarshalByRefObject
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we build an abstraction for TestSource? There seem to be multiple data associated with a test assembly and it is currently fragmented. E.g. we pass in source, maintain a state isDeploymentCompleted etc.. And now we're adding parallelization related configuration. These are effectively a per source property.

The TestSource could be serializable.

{
private ReflectHelper reflectHelper;

public TestAssemblySettingsProvider()
: this(new ReflectHelper())
{
}

internal TestAssemblySettingsProvider(ReflectHelper reflectHelper)
{
this.reflectHelper = reflectHelper;
}

public override object InitializeLifetimeService()
{
return null;
}

internal TestAssemblySettings GetSettings(string source)
{
var testAssemblySettings = new TestAssemblySettings();

// Load the source.
var testAssembly = PlatformServiceProvider.Instance.FileOperations.LoadAssembly(source, isReflectionOnly: false);

testAssemblySettings.ParallelLevel = this.reflectHelper.GetParallelizationLevel(testAssembly);

if (testAssemblySettings.ParallelLevel == 0)
{
testAssemblySettings.ParallelLevel = Environment.ProcessorCount;
}

testAssemblySettings.ParallelMode = this.reflectHelper.GetParallelizationMode(testAssembly);

testAssemblySettings.CanParallelizeAssembly = !this.reflectHelper.IsDoNotParallelizeSet(testAssembly);

return testAssemblySettings;
}
}
}
25 changes: 17 additions & 8 deletions src/Adapter/MSTest.CoreAdapter/Execution/TestExecutionManager.cs
Original file line number Diff line number Diff line change
@@ -230,21 +230,30 @@ private void ExecuteTestsInSource(IEnumerable<TestCase> tests, IRunContext runCo
testsToRun = tests.Where(t => MatchTestFilter(filterExpression, t, this.TestMethodFilter));
}

// Create test sets for execution, we can execute them in parallel based on parallel settings
IEnumerable<IGrouping<bool, TestCase>> testsets = Enumerable.Empty<IGrouping<bool, TestCase>>();
var parallelLevel = PlatformServiceProvider.Instance.TestSource.GetParallelizationLevel(source);
if (parallelLevel < 0)
var sourceSettingsProvider = isolationHost.CreateInstanceForType(
typeof(TestAssemblySettingsProvider),
null) as TestAssemblySettingsProvider;

var sourceSettings = sourceSettingsProvider.GetSettings(source);
var parallelLevel = sourceSettings.ParallelLevel;
var parallelMode = sourceSettings.ParallelMode;

if (MSTestSettings.CurrentSettings.TestParallelizationLevel > 0)
{
// Todo: This needs to be set to processor count when set to 0 from runsettings.
// The runsetings value takes precedence over an assembly level setting. Reset the level.
parallelLevel = MSTestSettings.CurrentSettings.TestParallelizationLevel;
}

if (parallelLevel >= 0)
if (parallelLevel > 0 && sourceSettings.CanParallelizeAssembly)
{
// Parallelization is enabled. Let's do further classification for sets.
var parallelMode = PlatformServiceProvider.Instance.TestSource.GetParallelizationMode(source);
var logger = (IMessageLogger)frameworkHandle;
logger.SendMessage(TestMessageLevel.Informational, $"MSTestExecutor: Parallel Configuration. Level = {parallelLevel}, Mode = {parallelMode}");
logger.SendMessage(
TestMessageLevel.Informational,
string.Format(CultureInfo.CurrentCulture, Resource.TestParallelizationBanner, parallelLevel, parallelMode));

// Create test sets for execution, we can execute them in parallel based on parallel settings
IEnumerable<IGrouping<bool, TestCase>> testsets = Enumerable.Empty<IGrouping<bool, TestCase>>();

// Parallel and not parallel sets.
testsets = testsToRun.GroupBy(t => t.GetPropertyValue<bool>(TestAdapter.Constants.DoNotParallelizeProperty, false));
45 changes: 45 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Helpers/ReflectHelper.cs
Original file line number Diff line number Diff line change
@@ -330,6 +330,41 @@ internal virtual string[] GetCategories(MemberInfo categoryAttributeProvider)
return testCategories.ToArray();
}

/// <summary>
/// Gets the parallelization level set on an assembly.
/// </summary>
/// <param name="assembly"> The test asembly. </param>
/// <returns> The parallelization level if set. -1 otherwise. </returns>
internal int GetParallelizationLevel(Assembly assembly)
{
var parallelizationLevelAttribute = PlatformServiceProvider.Instance.ReflectionOperations.GetCustomAttributes(assembly, typeof(TestParallelizationLevelAttribute)).OfType<TestParallelizationLevelAttribute>().FirstOrDefault();

if (parallelizationLevelAttribute != null)
{
return parallelizationLevelAttribute.ParallelizationLevel;
}

return -1;
}

/// <summary>
/// Gets the parallelization mode set on an assembly.
/// </summary>
/// <param name="assembly"> The test assembly. </param>
/// <returns> The parallelization mode for this assembly if set. TestParallelizationMode.MethodLevel if not.</returns>
internal TestParallelizationMode GetParallelizationMode(Assembly assembly)
{
var parallelizationModeAttribute = PlatformServiceProvider.Instance.ReflectionOperations.GetCustomAttributes(assembly, typeof(TestParallelizationModeAttribute)).OfType<TestParallelizationModeAttribute>().FirstOrDefault();

if (parallelizationModeAttribute != null)
{
return parallelizationModeAttribute.TestParallelizationMode;
}

// Default to highest degree of parallelization - Method Level.
return TestParallelizationMode.MethodLevel;
}

/// <summary>
/// Get the parallelization behavior for a test method.
/// </summary>
@@ -341,6 +376,16 @@ internal bool IsDoNotParallelizeSet(MemberInfo testMethod)
|| this.GetCustomAttributes(testMethod.DeclaringType.GetTypeInfo(), typeof(DoNotParallelizeAttribute)).Any();
}

/// <summary>
/// Get the parallelization behavior for a test assembly.
/// </summary>
/// <param name="assembly">The test assembly.</param>
/// <returns>True if test assembly should not run in parallel.</returns>
internal bool IsDoNotParallelizeSet(Assembly assembly)
{
return PlatformServiceProvider.Instance.ReflectionOperations.GetCustomAttributes(assembly, typeof(DoNotParallelizeAttribute)).Any();
}

/// <summary>
/// Gets custom attributes at the class and assembly for a method.
/// </summary>
2 changes: 2 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/MSTest.CoreAdapter.csproj
Original file line number Diff line number Diff line change
@@ -49,6 +49,8 @@
<Compile Include="Execution\StackTraceHelper.cs" />
<Compile Include="Execution\TestAssemblyInfo.cs" />
<Compile Include="Execution\TestClassInfo.cs" />
<Compile Include="Execution\TestAssemblySettingsProvider.cs" />
<Compile Include="ObjectModel\TestAssemblySettings.cs" />
<Compile Include="ObjectModel\TestFailedException.cs" />
<Compile Include="TestMethodFilter.cs" />
<Compile Include="Execution\TestMethodRunner.cs" />
9 changes: 8 additions & 1 deletion src/Adapter/MSTest.CoreAdapter/MSTestSettings.cs
Original file line number Diff line number Diff line change
@@ -332,7 +332,14 @@ private static MSTestSettings ToSettings(XmlReader reader)
{
if (int.TryParse(reader.ReadInnerXml(), out int parallelLevel))
{
settings.TestParallelizationLevel = parallelLevel;
if (parallelLevel == 0)
{
settings.TestParallelizationLevel = Environment.ProcessorCount;
}
else
{
settings.TestParallelizationLevel = parallelLevel;
}
}

break;
27 changes: 27 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/ObjectModel/TestAssemblySettings.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.

namespace Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel
{
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;

[Serializable]
internal class TestAssemblySettings
{
/// <summary>
/// Gets or sets the parallelization level.
/// </summary>
internal int ParallelLevel { get; set; }

/// <summary>
/// Gets or sets the mode of parallelization.
/// </summary>
internal TestParallelizationMode ParallelMode { get; set; }

/// <summary>
/// Gets or sets a value indicating whether the assembly can be parallelized.
/// </summary>
internal bool CanParallelizeAssembly { get; set; }
}
}
11 changes: 10 additions & 1 deletion src/Adapter/MSTest.CoreAdapter/Resources/Resource.Designer.cs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/Resource.resx
Original file line number Diff line number Diff line change
@@ -299,4 +299,7 @@
<data name="DiscoveryWarning" xml:space="preserve">
<value>[MSTest][Discovery][{0}] {1}</value>
</data>
<data name="TestParallelizationBanner" xml:space="preserve">
<value>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</value>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update string before merge. It may not have MSTest Executor.

</data>
</root>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.cs.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.de.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.es.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.fr.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.it.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.ja.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.ko.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.pl.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.ru.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.tr.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
5 changes: 5 additions & 0 deletions src/Adapter/MSTest.CoreAdapter/Resources/xlf/Resource.xlf
Original file line number Diff line number Diff line change
@@ -306,6 +306,11 @@
<target state="new">[MSTest][Discovery][{0}] {1}</target>
<note></note>
</trans-unit>
<trans-unit id="TestParallelizationBanner">
<source>MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</source>
<target state="new">MSTest Executor: Test Parallelization enabled. Parallel Level {0}, Parallel Mode {1}</target>
<note></note>
</trans-unit>
</body>
</file>
</xliff>
Loading