NUnit Console Extensions are not loaded in version 3.19

[vmayushan]

Hello!

I’ve encountered an issue with NUnit Console Runner version 3.19.0. The extensions (addins) are not being loaded as they were in version 3.18.3.

How to reproduce:

1. Download NUnit.Console-3.19.0.zip
2. Go to bin\net462
3. Run bin\net462>nunit3-console.exe --list-extensions

The output is

C:\Users\...\NUnit.Console-3.19.0\bin\net462>nunit3-console.exe --list-extensions
NUnit Console Runner 3.19.0 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Wednesday, January 8, 2025 6:08:49 PM
 
Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000
 
Installed Extensions
  Extension Point: /NUnit/Engine/NUnitV2Driver
  Extension Point: /NUnit/Engine/TypeExtensions/IService
  Extension Point: /NUnit/Engine/TypeExtensions/ITestEventListener
  Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
  Extension Point: /NUnit/Engine/TypeExtensions/IProjectLoader
  Extension Point: /NUnit/Engine/TypeExtensions/IResultWriter

Expected output is like it was in the 3.18.3

C:\Users\...\NUnit.Console-3.18.3\bin\net462>nunit3-console.exe --list-extensions
NUnit Console Runner 3.18.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Wednesday, January 8, 2025 6:08:23 PM
 
Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000
 
Installed Extensions
  Extension Point: /NUnit/Engine/NUnitV2Driver
    Extension: NUnit.Engine.Drivers.NUnit2FrameworkDriver(.NET 2.0)
      Version: 3.9.0.0
      Path: C:\Users\TeamCity Guest\vlmaiush\NUnit.Console-3.18.3\bin\net462\addins\nunit.v2.driver.dll
  Extension Point: /NUnit/Engine/TypeExtensions/IService
  Extension Point: /NUnit/Engine/TypeExtensions/ITestEventListener
    Extension: NUnit.Engine.Listeners.TeamCityEventListener(.NET 2.0) (Disabled)
      Version: 1.0.7.0
      Path: C:\Users\TeamCity Guest\vlmaiush\NUnit.Console-3.18.3\bin\net462\addins\teamcity-event-listener.dll
  Extension Point: /NUnit/Engine/TypeExtensions/IDriverFactory
  Extension Point: /NUnit/Engine/TypeExtensions/IProjectLoader
    Extension: NUnit.Engine.Services.ProjectLoaders.NUnitProjectLoader(.NET 4.0)
      Version: 3.7.0.0
      Path: C:\Users\TeamCity Guest\vlmaiush\NUnit.Console-3.18.3\bin\net462\addins\nunit-project-loader.dll
      FileExtension: .nunit
    Extension: NUnit.Engine.Services.ProjectLoaders.VisualStudioProjectLoader(.NET 2.0)
      Version: 3.9.0.0
      Path: C:\Users\TeamCity Guest\vlmaiush\NUnit.Console-3.18.3\bin\net462\addins\vs-project-loader.dll
      FileExtension: .sln .csproj .vbproj .vjsproj .vcproj .fsproj
  Extension Point: /NUnit/Engine/TypeExtensions/IResultWriter
    Extension: NUnit.Engine.Addins.NUnit2XmlResultWriter(.NET 4.0)
      Version: 3.6.0.0
      Path: C:\Users\TeamCity Guest\vlmaiush\NUnit.Console-3.18.3\bin\net462\addins\nunit-v2-result-writer.dll
      Format: nunit2

Thank you in advance for your help!

[Quppa]

Thanks for opening this issue - I'd been meaning to but didn't have time. Related discussion here: #1507 (comment)

[vmayushan]

@CharliePoole This is the way we use it in TeamCity: server downloads *.zip from GitHub, distributes it across build agents and agents are running nunit3-console.exe without additional modifications.

We can adjust the process if you guide us, maybe setting some env var will help?

[CharliePoole]

I've accepted this as a critical bug. Confirmation should be easy from the steps you provided. We define "Critical" to mean that an immediate release is needed to fix it.

@CharliePoole This is the way we use it in TeamCity: server downloads *.zip from GitHub, distributes it across build agents and agents are running nunit3-console.exe without additional modifications.

Thank you! You may be surprised to know that your single sentence is the most information I have ever received from JetBrains about how the console runner is installed under TeamCity. For 3.19, I had originally dropped the zip entirely from the distiribution, but I had a vague memory that it was being used by you so I restored it. It's not clear to me how the zip file package passed it's tests before release, but maybe I skpped a step in putting it back.

We can adjust the process if you guide us, maybe setting some env var will help?

Until about 2010, I had pretty regular contact with several JetBrains guys and until five years ago the teamcity extension was maintained by one of them. I no longer have any contacts. So, I'd say that the first need is for JetBrains to re-establish some kind of contact. Of course, it is possible that some of the users who have filed issues are actually from JetBrains, but nobody has identified themselves so far, except you. :-)

We already have an environment variable that tells us we are running under TeamCity. The original discussion on #1507 was because we silently fail if it is set but the extension is not installed. The discussion around fixing that led me to realize that the install process might not be working.

Version 3.19 had a very significant change in how extensions are located by the engine. My tests determined it to be non-breaking but of course that's only in the light of use cases I know about. Back when there was a primary JetBrains person working on the extension. I would have asked him to review the changes. So... a primary contact for the extension would be the biggest thing we need, both to receive and give guidance about how to keep this working!

Beyond this issue, there are a few other things that may need to be dealt with.

1. We are continuing to bundle version 1.0.7 of the extension. The latest version, 1.0.9, had a strange change... it continues to build for both .NET 2.0 and .NET Standard 2.0, but only packages the .NET 2.0 build. I didn't know what was intended and the release notes gave no information, so I stuck with bundling 1.0.7. If there are bug fixes in 1.0.9 you want to have released with the the runner, somebody needs to clarify this and we could update if needed for 3.20.

2. I'd prefer to remove the zip package, for reasons stated in Eliminate zip package in version 4 #1582. It has already been removed for version 4 and I'd do it for 3.20 if you guys could use the NUnit.Console nuget package in it's place. That seems like a fairly trivial change but if it isn't so for any reason, we can discuss it further.

3. For the past few years, the plan for version 4 of the runner has been to archive the teamcity extension and stop bundling it. Obviously, if we do that, JetBrains will need to do something. The JetBrains maintainer of the extension and I did have a plan for moving forward, but it was rejected by the company, so I don't know what will happen.

Anyway, I'll check out this bug. Feel free to contact me via email if you'd like to discuss any of this privately. I don't want to be more specific about past discussions or the people involved in a public forum.

FYI @OsirisTerje @NikolayPianikov

[CharliePoole]

@vmayushan I confirm this bug and I'll work on a fix.

As a temporary workaround, you can install the NUnit.Console nuget package into the same directory in which you unzipped the package. The extensions installed alongside the bin directory will be discovered.

As a further step, you could create your own zip package from this structure, possibly eliminating the extra copy of the console runner and the addins subdirectory.

[CharliePoole]

Ironically, the workaround explains why my package tests passed and allowed the release to go ahead. It happens that the zip package tests are run last of all the package types. The extensions installed by the nuget tests are already in place and the new algorithm used in 3.19 finds them. Running the zip package tests alone shows the problem.

[CharliePoole]

@vmayushan I'm planning to resolve this by changing the format of the zip file. I need you to verify that the specific internal changes won't affect your usage.

CURRENT: Extensions are stored in an addins directory. It's a flat directory containing all the extension dlls and their dependencies.

PROPOSED: Extensions will be stored in multiple directories, siblings to the bin directory. Their names will be of the form NUnit.Extension.EXTENSION.VERSION, which is how we do it for all other package types. I'll keep the executable path for the runner as bin\nunit3-console.exe for your usage.

RATIONALE: The new algorithm for finding "standard" extensions depends on this naming pattern. I'd prefer to allow that algorithm to work for the zip so long as we retain it. The alternative would be adding adhoc code to also look for a .addins directory, which I'd prefer to avoid. I assume that you are not directly accessing the extension directories in any way, but I'd like to be sure.

FYI: this investigation has uncovered a problem with the new --extensionDirectory command-line option, for which I'll create a new issue. If that were working properly, then I would have suggested you use it as a workaround. It turns out that the testing issue, which masked the installation problem also masked the problems with this option!

[CharliePoole]

@vmayushan I believe this is fixed in the 3.19.1-alpha.1 build on our myget feed. I'd appreciate your trying it before I do the full release.

[CharliePoole]

@vmayushan @Quppa I rushed this out as it seemed critical to you but there hasn't been any feedback. I don't want to release it as 3.19.1 until somebody tests it in a practical setting, running under TeamCity.

[vmayushan]

Hi @CharliePoole ! I am sorry for delayed verification, i am confirming that 3.19.1-alpha.1 works in TeamCity

Although I had an issue with using package from MyGet: https://www.myget.org/feed/nunit/package/nuget/NUnit.ConsoleRunner
Looks like it doesn't bundle any addins. It might be I have downloaded wrong package though...

To verify 3.19.1 I have cloned repository with corresponding tag and built it with build.cmd --target=BuildZipPackage and uploaded it to TC.

Not related to the task: we have tried to reach you in NUnit slack, please check it when you have time.

Thank you again for the quick fix!

[CharliePoole]

This issue has been resolved in version 3.19.1

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org