An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing ')'.

[jerryk414]

ENVIRONMENT

---

NUnit: 3.10.1
NUnit3TestAdapter: 3.10.0
Visual Studio: 15.9.2
.NET Platform: .NET Core 2.1

DESCRIPTION

---

When executing a category of tests, if a test is included in the list of tests that contains an opening parenthesis, but does not contain the closing parenthesis, the following error may occur, and the test run will fail:

An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing ')'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

REPRODUCTION

---

This issue can only be reproduced when selecting a group of tests to run - selecting individual tests, or selecting 'Run All' will no reproduce the issue.

It appears that reproducing the issue requires there to be a test that includes an opening parenthesis but does not include a closing parenthesis, and a test that ends with the closing bracket character (']').

You should be able to reproduce the issue by using the following test code:

public class TestNameRepro
{
    #region TestCaseSource

    protected static IEnumerable TestName_TestCases
    {
        get
        {
            // Adding a parenthesis to the end of this test name will stop the exception from throwing (e.g. $"TestName(...)")
            yield return new TestCaseData(1).SetName($"TestName(...");

            // Cannot be duplicated without a second test included that ends with a ']'
            yield return new TestCaseData(2).SetName($"TestName(...)]");
        }
    }

    #endregion     
    [Test, TestCaseSource(nameof(TestName_TestCases))]
    public void TestName(int input)
    {
        // Irrelevant  
    }
}

Once built, categorize the test explorer using the 'Class' category selection, and select the test suite 'TestNameRepro' and click 'Run Selected Tests':

The 'Tests' output window should display the error mentioned above:

An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing ')'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

[OsirisTerje]

Sorry for being late here.
You're right on this one, I can confirm/repro it.

Question is if these characters should be filtered off, which is the safest way, I would assume.

[Dragonsangel]

I am hanging this one onto this bug report since I think it would be in the same area.

When you have a typeof with a Type that contains a Tuple inside a TestCase (example [TestCase(typeof(IEnumerable<(String oneValue, Int32 twoValue)>))]), then the adapter throws

[Error] An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

This error only occurs when you try to perform a "Run all tests in Class" or "Run all tests".
When you run the test on its own, then it is executed.

Repo Test:

[Test]
[TestCase(typeof(IEnumerable<(String oneValue, Int32 twoValue)>))]
public void UnitTest_TestCaseWithTuple_TestIsNotExecuted(Type targetType)
{
	Assert.That(targetType == typeof(IEnumerable<(String oneValue, Int32 twoValue)>), Is.True);
}

[gavinschultz]

I think @Dragonsangel is correct that tuples cause related issues, due to the parsing of the embedded parentheses.

There likewise seem to be issues for any tuples passed in as test case data, due to their string representation always including parentheses. This code, which in the test runner would generate a name of TestA((1, 2), 5), does not run:

[Test, TestCaseSource(nameof(SourceA))]
public void TestA((int a, int b) x, int y) { }
public static IEnumerable SourceA =>
    new[] { new TestCaseData((a: 1, b: 2), 5) };

If the tuple is the final parameter as below, the test name is TestB(5, (1, 2)). This runs okay, perhaps because the generated test name has no further characters after the tuple's closing parenthesis.

[Test, TestCaseSource(nameof(SourceB))]
public void TestB(int y, (int a, int b) x) { }
public static IEnumerable SourceB =>
    new[] { new TestCaseData(5, (a: 1, b: 2)) };

One can also work around the problem by overriding the parameter names with SetArgDisplayNames() (though for some reason I didn't have as much luck working around it via SetName(). This works:

[Test, TestCaseSource(nameof(SourceC))]
public void TestC((int a, int b) x, int y) { }
public static IEnumerable SourceC =>
    new[] { new TestCaseData((a: 1, b: 2), 5).SetArgDisplayNames("a+b", "y") };

[johnjaylward]

Using Visual Studio 2019 (v16.2.3) I'm unable to run any tests. I get the following error:

[8/29/2019 11:43:00.708 AM Informational] NUnit Adapter 3.15.0.0: Test execution started
[8/29/2019 11:43:00.716 AM Error] An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

This happens every time, no matter how I select to run the tests (run all, run selected, etc...)

[johnjaylward]

As mentioned in #650, I tried downgrading (I uninstalled the VSIX adapter) and set my projects to use the 3.14 package. somehow, it still looks like it's choosing the wrong version (3.13 somehow?), but it's not working:

[8/29/2019 12:05:42.828 PM Informational] NUnit Adapter 3.13.0.0: Test execution started
[8/29/2019 12:05:42.833 PM Error] An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

Any other ideas I can try to get my tests working?

[johnjaylward]

Looks like downgrading to VS 2017 (v15.9.15) lets me run my tests. Whatever issue I'm having is related specifically to how 2019 is calling the adapter it seems.

Output using VS2017 (slightly truncated):

[8/29/2019 12:31:20 Informational] ------ Discover test started ------
[8/29/2019 12:31:20 Informational] NUnit Adapter 3.13.0.0: Test discovery starting
[8/29/2019 12:31:21 Informational] NUnit Adapter 3.13.0.0: Test discovery complete
...
[8/29/2019 12:31:23 Informational] ========== Discover test finished: 57 found (0:00:03.1097103) ==========
[8/29/2019 12:31:23 Informational] ------ Run test started ------
[8/29/2019 12:31:23 Informational] NUnit Adapter 3.13.0.0: Test execution started
[8/29/2019 12:31:23 Informational] Running all tests in My.Tests.dll
[8/29/2019 12:31:24 Informational]    NUnit3TestExecutor converted 1 of 1 NUnit test cases
[8/29/2019 12:31:25 Informational] Running all tests in My2.Tests.dll
[8/29/2019 12:31:25 Informational]    NUnit3TestExecutor converted 56 of 56 NUnit test cases
[8/29/2019 12:31:26 Informational] NUnit Adapter 3.13.0.0: Test execution complete
[8/29/2019 12:31:26 Informational] ========== Run test finished: 56 run (0:00:02.7695974) ==========

[OsirisTerje]

@johnjaylward Do you have any simple code that can repro this behaviour ?

[johnjaylward]

I'm using test cases sources that look similar to what's in #650 .
I usually use the nameof keyword for the method names though:

[Test(), TestCaseSource(typeof(CaseTestData), nameof(CaseTestData.EqualsData))]
public void EqualsTest(Case case1, Case case2)
{
    Assert.AreEqual(case1, case2);
}

most of my tests in this project are very simple like that.

[Benoit-Ballet-Clario]

On my side, I have the same issue (on VS2019 last update) by adding this testCase :
[TestCase("invalid-characters-in-sld@!\"#$%(),/;<>_[n")]
If I just remove one component of the String I don't have the issue anymore.
Weird

[johnjaylward]

@OsirisTerje , I think it may be due to me testing lamdba expressions on a custom Linq.IQueryProvider. Here's my test case data. I'm pretty sure it was working before I started testing my Linq items:

using NUnit.Framework;
using REST.API.DAL;
using REST.API.Model;
using REST.API.Linq;
using System;
using System.Collections.Generic;
using System.Linq.Expressions;
using System.Linq;
using Moq;

namespace REST.API.Tests.Linq
{
    public static class ExpressionTransatorTestData
    {
        public static IEnumerable<TestCaseData> TranslateData
        {
            get
            {
                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.Approval == "test"),
                    "approval=test"
                    );

                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.AutoCreatedCase == false),
                    "auto_created_case=false"
                    );

                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.AutoCreatedCase == true),
                    "auto_created_case=true"
                    );

                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.Approval == "test" && c.AutoCreatedCase == false),
                    "approval=test^auto_created_case=false"
                    );

                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.Approval == "test" || c.AutoCreatedCase == false),
                    "approval=test^ORauto_created_case=false"
                    );

                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.Approval == "test" || c.AutoCreatedCase == true),
                    "approval=test^ORauto_created_case=true"
                    );

                // Mon Jan 2 15:04:05 MST 2006
                // yyyy-MM-dd HH:mm:ss
                DateTimeOffset testDateTime = new DateTimeOffset(2006, 1, 2, 3, 4, 5, TimeSpan.FromHours(-7));

                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.Approval == "test"
                        && c.AutoCreatedCase
                        || c.ActivityDue < testDateTime),
                    "approval=test^auto_created_case^ORactivity_due<2006-01-02+10%3A04%3A05"
                    );

                // test using a "new DateTime" in the expression
                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.Approval == "test"
                        && c.AutoCreatedCase
                        || c.ActivityDue < new DateTimeOffset(2006, 1, 2, 3, 4, 5, TimeSpan.FromHours(-7))),
                    "approval=test^auto_created_case^ORactivity_due<2006-01-02+10%3A04%3A05"
                    );

                // test using a "new DateTime" in the expression
                yield return new TestCaseData(
                    (Expression<Func<Case, bool>>)((Case c) => c.Approval == "test"
                        && c.AutoCreatedCase
                        || c.ActivityDue < new DateTimeOffset(2006, 1, 2, 3, 4, 5, TimeSpan.FromHours(-7))),
                    "approval=test^auto_created_case^ORactivity_due<2006-01-02+10%3A04%3A05"
                    );


                var mockAPI = new Mock<IRestApi>();
                mockAPI.Setup(r => r.CreateQuery(It.IsAny<Expression>())).Returns<Expression>((e) =>
                {
                    return (IQueryable)Activator.CreateInstance(typeof(Query<>).MakeGenericType(e.Type), new object[] { mockAPI.Object, e });
                });
                mockAPI.Setup(r => r.CreateQuery<Case>(It.IsAny<Expression>())).Returns<Expression>((e) =>
                {
                    return new Query<Case>(mockAPI.Object, e);
                });
                mockAPI.Setup(r => r.CreateQuery<Case>(It.IsAny<Expression<Func<Case,bool>>>())).Returns<Expression<Func<Case, bool>>>((e) =>
                {
                    return new Query<Case>(mockAPI.Object, e);
                });

                Query<Case> query = new Query<Case>(mockAPI.Object);

                yield return new TestCaseData(
                    query.Where(c => c.Approval == "test").Expression,
                    "approval=test"
                    );

                // test ref-link converions
                yield return new TestCaseData(
                    query.Where(c => c.Account == "86837a386f0331003b3c498f5d3ee4ca").Expression,
                    "account=86837a386f0331003b3c498f5d3ee4ca"
                    );

                Guid boxeoGuid = Guid.ParseExact("86837a386f0331003b3c498f5d3ee4ca", "N");
                yield return new TestCaseData(
                    query.Where(c => c.Account == boxeoGuid).Expression,
                    "account=86837a386f0331003b3c498f5d3ee4ca"
                    );

                //Check order by
                yield return new TestCaseData(
                    query.Where(c => c.Approval == "test").OrderBy(c => c.Number).Expression,
                    "approval=test^ORDERBYnumber"
                    );

                yield return new TestCaseData(
                    query.Where(c => c.Approval == "test").OrderBy(c => c.Number).ThenByDescending(c => c.Category)
                        .Expression,
                    "approval=test^ORDERBYnumber^ORDERBYDESCcategory"
                    );

                // check multiple where
                yield return new TestCaseData(
                   query.Where(c => c.Approval == "test").Where(c => c.Number != "5").Expression,
                   "approval=test^number!=5"
                   );

                // test dot walks
                yield return new TestCaseData(
                    query.Where(c => (+c.Account).Name == "boxeo").Expression,
                    "account.name=boxeo"
                    );

                yield return new TestCaseData(
                    query.Where(c => ((Account)c.Account).Name == "boxeo").Expression,
                    "account.name=boxeo"
                    );

                yield return new TestCaseData(
                    query.Where(c => (+(+c.Account).ParentAccount).Name == "boxeo").Expression,
                    "account.account_parent.name=boxeo"
                    );

                yield return new TestCaseData(
                    query.Where(c => ((Account)((Account)c.Account).ParentAccount).Name == "boxeo").Expression,
                    "account.account_parent.name=boxeo"
                    );

                yield return new TestCaseData(
                    query.Where(c => (+((Account)c.Account).ParentAccount).Name == "boxeo").Expression,
                    "account.account_parent.name=boxeo"
                    );

                yield return new TestCaseData(
                    query.Where(c => ((Account)(+c.Account).ParentAccount).Name == "boxeo").Expression,
                    "account.account_parent.name=boxeo"
                    );


                yield return new TestCaseData(
                    query.Where(c => (+(+(+c.Account).ParentAccount).ParentAccount).Name == "boxeo").Expression,
                    "account.account_parent.account_parent.name=boxeo"
                    );

                yield return new TestCaseData(
                    query.Where(c => (+(+(+c.Account).ParentAccount).ParentAccount).Name == "boxeo"
                        && c.Active == true
                        || c.State == CaseState.New
                    ).Expression,
                    "account.account_parent.account_parent.name=boxeo^active=true^ORstate=1"
                    );
            }
        }
    }
}

[OsirisTerje]

@johnjaylward Can you upload this to a git repo somewhere, with the rest of the files, so it becomes a working compilable repro of this case ?
You have added this info to this issue, are you sure this is not a new issue?

[johnjaylward]

Not readily, I have not been authorized by my company to release the source code. I'll see if I can make time to create a sample project that is slimmed down to just the ideas

[johnjaylward]

I'm pretty sure it's the same issue, or at least very similar, as the error I get is as follows:

[9/17/2019 12:22:50.047 PM Error] An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

It would be more helpful if the error message included the test name/ data that was causing the issue.

[johnjaylward]

@OsirisTerje , here is a sample solution that reproduces the issue I see.

https://github.com/johnjaylward/nUnitExpressionTest

[johnmwright]

@OsirisTerje is there anything I can do to help with this issue? I'm happy to spend some time on it and submit a PR if you could give me a bit of guidance on where to focus.

For a bit of reference, I have several tests that hit this issue, mostly due to mismatched ", ( or ) chars in the generated name using .SetName().

For example:
.SetName("Prod exception 'pp:-) :-) xzmmmmmjn'")
.SetName("Prod exception '\"c'")

[jnm2]

It seems like the adapter is not escaping special characters when creating a filter from the test names that VS passes in (which originated from these SetName calls).

[OsirisTerje]

The second example line above is already escaped, and if I got it right, that one also fails. Or do you mean something else?

[OsirisTerje]

@johnmwright It would be awesome if you could step in here, any contributions are really appreciated. That said, this issue might not be easy, since the exception comes from the testhost, which doesn't accept our test names.

The crash happens here:

nunit3-vs-adapter/src/NUnitTestAdapter/TfsTestFilter.cs

Line 104 in bddee2c

get { return TfsTestCaseFilterExpression == null || TfsTestCaseFilterExpression.TestCaseFilterValue == string.Empty; }

It is caused by the special characters that is included in the test names, and which is not accepted by the FQN "standard" as set by the Visual Studio test host. Escaping them should be a way out, but I am not sure.

I think what is needed is to get more information on what works and what doesn't here.

You can build your own debug package and enable breakpoints by turning on the symbol here (uncomment it):

nunit3-vs-adapter/src/NUnitTestAdapter/NUnit3TestExecutor.cs

Line 24 in bddee2c

// #define LAUNCHDEBUGGER

You build the package on command line by:
build -c debug
build -c debug -t package

The generated package can then be found in the package folder.
Also, step up the version number here

nunit3-vs-adapter/build.cake

Line 15 in bddee2c

var version = "3.15.1";

, set it to 3.16.0
Add a number to the modifier below, like ="01", and later step this one up for each new build you need to do

[jnm2]

The second example line above is already escaped, and if I got it right, that one also fails. Or do you mean something else?

If there is an error about a mismatched ", ( or ) chars, it seems like there must be some level where the correct escaping isn't happening.

[johnmwright]

@OsirisTerje good news! I was able to get some time this week to work on this and have a working solution. I need to clean it up a bit and run through some of the other cases reported above before filing a PR. Hoping to have something for you this weekend.

Any requests/requirements you need from me in order to submit PRs?

[OsirisTerje]

@johnmwright You should be good to go.

Notes:

1. Do not change anything but the code you actually are fixing.
2. If you do anything that may have side effects, or if you are just a bit unsure if it does, please discuss the changes. You can hold a PR by adding a line at the top like *** DO NOT MERGE ***, then we know it is "work in progress", and you can use the PR as a discussion "forum". You can also discuss your suggestions here if you prefer.
3. Link your PR to this issue :-)

[johnmwright]

FYI: I have filed PR #668 for this issue.

[OsirisTerje]

@johnjaylward Can you confirm your code works with the changes in PR #668 ?
I checked with your code, and could not make it work, but I think this is in your code and not in the fix. You throw in the Translate method. Would be awesome if you could confirm, that it otherwise works as it should for you.

A dev package should appear within a few minutes now at https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter , version 3.16.0-dev-*

[johnjaylward]

So, It solves the issue with the test project I made, but not the main project with my actual code. I've pushed the changes up to my test repo to have the tests pass instead of fail, and also updated to use the dev version of the test adapter. This change definitely solved some of the problems though.

I'll need to figure out what's causing the issue in my main project. It seems to be separate from the naming that got fixed there.

[johnjaylward]

@OsirisTerje I'm going to open a new ticket for the new behavior I'm seeing.

[OsirisTerje]

@johnjaylward Thanks, that's the best way to handle it too :-) If this is something with the adapter, it would be good to have it in. I would like to see whatever that is before we do a new release.

[OsirisTerje]

Dev package resolving this is at https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/3.16.0-dev-01202

[Dragonsangel]

As an update to this, I tested the fix with the 3.16.0-dev-01202 package and this Test is not run with the same error message as described in this issue.

Error] An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed.

public class TupleUnitTests
{
	[Test]
	[TestCase(typeof(IDummy<(String body, Dictionary<String, String> applicationData), String>))]
	public void UnitTest_TestCaseWithTuple_TestIsNotExecuted(Type targetType)
	{
		Assert.That(targetType, Is.Not.Null);
	}
}

public interface IDummy<T1, T2> : IList<T1> { }

If I however switch around the Types, then it will execute
[TestCase(typeof(IDummy<String, (String body, Dictionary<String, String> applicationData)>))]

Summary:
TestCase[Type<Tuple(), Type>] Error
TestCase[Type<Type, Tuple()>] Executed

[goamn]

Happened to me on VS2022. Fixed it by removing the "{m}" modifier inside of the property TestName.

[danielcoppa-olo]

Occurring for me on VS2022, but not VS2019. The project will throw the "missing (" exception when a test group is started from Test Explorer in VS2022, but opening the same project in VS2019 and running the same test group will succeed.