Skip to content
This repository has been archived by the owner on Jun 12, 2021. It is now read-only.

Switch to xUnit.net 2.0 #51

Closed
35 tasks done
adamralph opened this issue May 2, 2013 · 40 comments
Closed
35 tasks done

Switch to xUnit.net 2.0 #51

adamralph opened this issue May 2, 2013 · 40 comments
Labels
breaking This change could break current consumers enhancement New feature or request
Milestone

Comments

@adamralph
Copy link
Owner

This is an opinionated subset of 1.0 features which I believe are required in version 2. If any features have been omitted, it is because I have never used them, nor have I heard of anybody using them, or I have only heard of them being used in an edge case. Ultimately the framework is designed to be extensible so if any edge cases are not being catered for then the extensibility points should be examined first before adding specific features.

This issue only tackles net45 support. Support for other platforms has been spun off into #220, #221, #222, #223 and #224.

  • Update to stable xunit 2.0 release
  • Features
    • AsyncScenarioFeature
    • AsyncStepFeature
    • BackgroundFeature
    • DefaultParametersFeature
    • ExampleFeature
    • ObjectDisposalFeature
      • Remove teardown step and send ITestCaseCleanupFailure if required (2.0 change)
    • ScenarioFeature
    • SkippedScenarioFeature
    • SkippedStepFeature
    • TeardownFeature
      • Remove teardown step and send ITestCaseCleanupFailure if required
  • xUnit.net 2.0 feature support
    • BeforeAfterTestFeature - before after scenario instead of step
    • ClassFixtureFeature
    • CollectionFixtureFeature
    • MemberDataFeature
    • TestClassFeature (was designed as an xbehave 1.x feature but not tested and did not work)
    • TraitsFeature
  • Build
    • Build packages
  • Runner compatibility
    • xunit.runner.console
    • xunit.runner.visualstudio
    • xunit.runner.msbuild
    • TestDriven.net
    • Resharper (thanks to @citizenmatt for debugging)
      • extra 'aborted' test
      • a skipped step leaves the entire scenario marked as 'inconclusive'
      • a skipped scenario results in thousands of redundant nodes in the UI
    • NCrunch
      • scenarios are not discovered (successfully tested in debug build provided by @remcomulder)
      • skipped steps cause scenario to show as failed (https://gist.github.com/adamralph/110fb590a14c8cda41ff) needs a new build of NCrunch which reports the tests as inconclusive, so that the user can configure whether to show them as failed (default) or passed
@adamralph adamralph changed the title Add 2.x alpha project Convert net45 target to xunit 2.0 Apr 18, 2014
@adamralph adamralph self-assigned this Apr 18, 2014
adamralph added a commit that referenced this issue Apr 28, 2014
#51 refactor: added v2 acceptance tests, minimal v2 impl
@adamralph adamralph mentioned this issue Jul 8, 2014
@adamralph adamralph changed the title Convert net45 target to xunit 2.0 Convert to xunit 2.0 Jul 8, 2014
adamralph added a commit that referenced this issue Jul 8, 2014
@adamralph adamralph added this to the 2.0 milestone Jul 18, 2014
@ursenzler
Copy link
Contributor

I checked xBehave.Core. Looks good.

@adamralph
Copy link
Owner Author

Great, thanks for trying it!

With regard to the crazy idea, unfortunately xunit does not have the concept of an inconclusive test result. The only possible results are Failed, Passed and Skipped.

What we could do is have new version of ContinueOnFailureAfterAttribute which takes a regex pattern in its constructor. Any step which matches the regex can trigger the behaviour. E.g. to continue after failure of any step beginning with "Then... " and subsequent steps, you would write

[assembly: ContinueOnFailureAfter("^Then")]

@ursenzler
Copy link
Contributor

That would work for me.

@adamralph
Copy link
Owner Author

OK, let's get that into RC3.

@ursenzler
Copy link
Contributor

I migrated a lot of specs to xBehave 2.0. All works well, no problems found. Hope you get rid of the pre-release soon - so that my automatic scripts for updates work again :-)

@adamralph
Copy link
Owner Author

That's great! It certainly gives the RC a good vote of confidence.

@adamralph
Copy link
Owner Author

@ursenzler I've spent some good time agonising over the 'continue on failure' feature. What bothered me was that it's a highly specific extension and, especially, one that I've never used.

So I came up with a different idea. I've added a new step builder method, ContinueOnFailure() which can be used to indicate that a specific step should exhibit this behaviour, e.g.

[Scenario]
public void Something()
{
    "Given something"
        .f(() => ...);

    "When something"
        .f(() => ...);

    "Then something"
        .f(() => ...)
        .ContinueOnFailure();

    "And something else"
        .f(() => ...);

    "And something else again"
        .f(() => ...);
}

In this scenario, if "Then something" fails then the remaining "And..." steps are still executed instead of being skipped. However, if "And something else" fails then "And something else again" is skipped, since the ContinueOnFailure() only applies to "Then something".

This allows control of the behaviour at the most granular level possible, i.e. individual steps. This is a feature I think I might use 😉.

Now, in order to allow people to emulate the behaviour of the 1.x ContinueOnFailureAfterAttribute, I propose to add a 2.x StepDefinitionsFilterAttribute (name not finalised) as an extensibility point, e.g.

[Scenario]
[ContinueOnFailureAfterThen]
public void Something()
{
    "Given something"
        .f(() => ...);

    "When something"
        .f(() => ...);

    "Then something"
        .f(() => ...);

    "And something else"
        .f(() => ...);

    "And something else again"
        .f(() => ...);
}

// your own custom extension
public class ContinueOnFailureAfterThenAttribute : StepDefinitionsFilterAttribute
{
    public override void Filter(ICollection<IStepDefinition> steps)
    {
        foreach(var step in steps.SkipWhile(step => !step.Text.StartsWith("Then ")))
        {
            step.ContinueOnFailure = true;
        }
    }
}

Implementations of 2.x StepDefinitionsFilterAttribute can be added at the method, class or assembly level, similarly to the 1.x ContinueOnFailureAfterAttribute.

Thoughts?

@ursenzler
Copy link
Contributor

I like the idea with the attribute more. Mainly because I use "establish", "when", "it", "it" with no distinction between the assertion steps. The step builder method brings the concept of order to my assertion steps, which I don't like. What if I change the order?

The attribute would allow to run all "it" even if a single "it" fails. That's what I like.

@adamralph
Copy link
Owner Author

Which attribute are you referring to? There is the old 1.x attribute and the proposed new 2.x attribute.

@adamralph
Copy link
Owner Author

With 2.x attribute you can achieve what you want like so:

[Scenario]
[ContinueOnFailureAfterIt]
public void Something()
{
    "Establish something"
        .f(() => ...);

    "When something"
        .f(() => ...);

    "It something"
        .f(() => ...);

    "It something else"
        .f(() => ...);

    "It something else again"
        .f(() => ...);
}

// your own custom extension
public class ContinueOnFailureAfterItAttribute: StepDefinitionsFilterAttribute
{
    public override void Filter(ICollection<IStepDefinition> steps)
    {
        foreach(var step in steps.SkipWhile(step => !step.Text.StartsWith("It ")))
        {
            step.ContinueOnFailure = true;
        }
    }
}

You don't need to use the step builder method at all. And you can freely change the order of your It steps.

@ursenzler
Copy link
Contributor

I meant that the new proposed attribute solution is just perfect for my needs.
I won't need the step builder method.

@adamralph
Copy link
Owner Author

Ah, great! 😄

OK, so that will be part of RC3 - tracked by #254

@adamralph adamralph changed the title Switch to xunit 2.0 Switch to xUnit.net 2.0 Oct 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
breaking This change could break current consumers enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants