Perf tests are broken when run from project test directory

[stephentoub]

I'm trying to build and run the System.Collections perf tests. Following the instructions at https://github.com/dotnet/corefx/blob/master/Documentation/project-docs/performance-tests.md, I did a:

git clean -xdf
build -release

from the root of my repo, and then from src\System.Collections\tests\Performance did:

msbuild /t:BuildAndTest /p:Performance=true /p:ConfigurationGroup=Release /p:TargetOS=Windows_NT

and I get the error:

The application to execute does not exist: 'corefx\bin\tests\System.Collections.Performance.Tests\netcoreapp-Windows_NT-Release-x64\PerfRunner.exe'

[stephentoub]

cc: @mellinoe, this is the issue I mentioned to you.

[mellinoe]

I talked with @DrewScoggins about this a little bit at the end of the day. The tests work if you run them with the root script, but not if you use the command above. The CI tests use this command line, which Drew and I were able to use successfully:

build-managed.cmd -release -tests -- /p:Performance=true /p:TargetOS=${osGroup} /m:1

This is obviously not ideal (or even workable, really), considering how long this takes.

We didn't figure out exactly why building projects directly didn't work, but it had something to do with the tests building and running out of the wrong directory.

[stephentoub]

Thanks, @mellinoe and @DrewScoggins.

This is obviously not ideal (or even workable, really)

Yeah, this isn't viable. That's fine for doing a full run to see where we currently stand, but it essentially prohibits writing any new performance tests.
cc: @danmosemsft

[danmoseley]

@DrewScoggins @brianrob can someone take a look why this broke? Your team will need this, to write more tests.

[brianrob]

@DrewScoggins, can you please take a look at this?

[DrewScoggins]

Yeah, I have already started working on this.

[brianrob]

Thanks!

[stephentoub]

Thanks, @DrewScoggins .

[DrewScoggins]

Sorry, this has taken a couple of days. I have had some other stuff going on.

I looked into this and the problem seems to be happening in the GetBinPlaceConfiguration task. In that task we setup the BinPlaceDir ItemGroup, but only do that if we have a definition of _currentBinPlaceConfigurations. This is not being set because of this condition, Condition="'%(Identity)' == '$(Configuration)' OR '%(Identity)-$(ConfigurationGroup)' == '$(Configuration)'". When we run the CI the configuration property is getting set to Configuration = netcoreapp-Windows_NT-Release, but when we manually call the csproj it is set this way, Configuration = netcoreapp-Windows_NT-Release-x64. And if you look at the condition all of the Identity variables are of the form netcoreapp-Windows_NT and will never end up evaluating true.

I did some digging here, and could not come up with a reason why we were seeing this behavior. Does anyone else with a little better knowledge of the build system and MSBuild have an idea what is causing this?

[danmoseley]

@joperezr ?

[joperezr]

I'm pretty sure that this is caused by the recent changes that @safern made around changing the testPath location. The reason why I think that is that I tried this out locally and both the test assembly and the PerfRunner.exe where dropped to corefx\bin\Windows_NT.AnyCPU.Release\System.Collections.Performance.Tests\netcoreapp folder, which is what we used to have before his change. On the other hand, RunTests.cmd is located at corefx\bin\tests\System.Collections.Performance.Tests\netcoreapp-Windows_NT-Release-x64\ which is the path that gets set after his changes.

[safern]

Yes this looks like it’s affected by my change. I might have missed to include the bin placing configuration when running perf tests. I will take a look. Self assigned.

[mellinoe]

Thanks for looking into it! If there was something that recently changed the test paths, then I would definitely think that it is related.

[stephentoub]

@safern, any progress on this?

[safern]

@safern, any progress on this?

Yes I started taking a look yesterday afternoon and I'm close to get the fix. Will update you soon when I get it.

[safern]

Fix is out: dotnet/corefx#24493