Implement test and fixture modifiers

[inikulin]

Features summary

test.only, test.skip, fixture.only, fixture.skip (closes #246)

You can now skip specific fixtures or tests using .skip modifier. In contrast, you can run only specified tests or fixtures using .only directive.

.skip example:

fixture.skip `Fixture1`; // All tests in this fixture will be skipped

test('Fixture1Test1', () => {});
test('Fixture1Test2', () => {});

fixture `Fixture2`;

test('Fixture2Test1', () => {});
test.skip('Fixture2Test2', () => {}); // This test will be skipped
test('Fixture2Test3', () => {});

.only example:

fixture.only `Fixture1`; 
test('Fixture1Test1', () => {});
test('Fixture1Test2', () => {});

fixture `Fixture2`;

test('Fixture2Test1', () => {});
test.only('Fixture2Test2', () => {}); 
test('Fixture2Test3', () => {});

// Only tests in Fixture1 and Fixture2Test2 will run

Note that reporter plugin now accepts additional skipped field in testRunInfo object, thus plugin now can decorate skipped tests in the distinct manner. All built-in reporters and plugin generator were updated to support this feature.

test.before and test.after hooks (closes #1108)

Test now supports test.before and test.after hooks that behave the same way as fixture.beforeEach and fixture.afterEach hooks with the exception that hook will run only for the specified test. If fixture.beforeEach and fixture.afterEach hooks are specified as well, test-specific hook will override them.

Example:

fixture `Fixture1`
    .beforeEach(async t  => {...})   
    .afterEach(async t  => {...});

test
    .before(async t => {...}) // Overrides fixture.beforeEach    
   ('Test1', async t => {...});

fixture `Fixture2`;

test
    .after(async t => {...})
    .before(async t => {...})
    ('Test2', async t => {...});

Note for tech writers: Current fixture.beforeEach and fixture.afterEach hooks section in documentation is not quite accurate. It's worth mentioning that hooks are run for each test run of the given test. So e.g., if we have 3 browsers set for the test run, each hook will be executed in each browser right before and after the test.

t.ctx object (closes #841)

t.ctx is an object that is shared between fixture.beforeEach, fixture.afterEach, test.before, test.after hooks and test itself. Each test run has it's own t.ctx object. This object can be used to share variables and other test-specific stuff between test functions without introducing global variables (it's even more complicated in TestCafe since single test can be run in several browsers).

Example:

fixture `Fixture1`
    .beforeEach(async t  => {
          t.ctx.someVar = 123;
     })   
    .afterEach(async t  => {
          console.log(t.ctx.someVar); // > 123
    });

test
    .before(async t => {
         console.log(t.ctx.someVar); // > 123
    })  
   ('Test1', async t => {
        console.log(t.ctx.someVar); // > 123
   });

By default t.ctx is initialized to empty object without prototype (so it's safe to iterate its keys without hasOwnProperty check). However, you can assign anything to t.ctx and assigned value will be shared, e.g.:

t.ctx = 123;

test.page directive (closes #501)

You can now specify startup page for the particular test that will override page specified for the fixture. It was possible previously to navigate to the specific page in the test by calling t.navigateTo, but it had a drawback that test first navigated to the page specified for the fixture. Additional navigation now avoidable with test.page directive.

Example:

fixture `Some fixture`
    .page `http://page1`;

test('Test1', () => {}); // Will navigate to http://page1
test
    .page `http://page2` 
   ('Test2', () => {}); // Will navigate to http://page2

test.httpAuth directive (closes #1109)

In addition to newly introduced HTTP authentication feature (#955) it's now possible to specify authentication credentials for particular test. If credentials are specified for fixture as well, they will be overridden by test-specific credentials.

Example:

test
    .httpAuth({ username: 'hey', password: 'yo' })
    ('Some test', async t => {});

---

Note that you can specify fixture and test directives before and after declarations, so all this cases are valid:

fixture.skip `Fixture name` .beforeEach(() => {});

fixture
    .page `http://page`
    `Fixture name`;

test
   .before(() => {})  
   ('Test name', () => {})
   .after(() => {});

test('Some test', () => {}).skip;

\cc @helen-dikareva @AlexanderMoskovkin @churkin @AndreyBelym @lexkazakov @VasilyStrelyaev

[testcafe-build-bot]

❌ Tests for the commit cea63e1 have failed. See details:

• ✅ continuous-integration/appveyor/branch

• ❌ continuous-integration/travis-ci/push.

[testcafe-build-bot]

❌ Tests for the commit 3ebb00c have failed. See details:

• ❌ continuous-integration/appveyor/branch

• ❌ continuous-integration/travis-ci/push.

[testcafe-build-bot]

❌ Tests for the commit acc4799 have failed. See details:

• ✅ continuous-integration/appveyor/branch

• ❌ continuous-integration/travis-ci/push.

[testcafe-build-bot]

❌ Tests for the commit 4545474 have failed. See details:

• ❌ continuous-integration/appveyor/branch

• ❌ continuous-integration/travis-ci/push.

[testcafe-build-bot]

✅ Tests for the commit 1c44154 have passed. See details:

• ✅ continuous-integration/appveyor/branch

• ✅ continuous-integration/travis-ci/push.

[testcafe-build-bot]

❌ Tests for the commit 84c88a2 have failed. See details:

• ❌ continuous-integration/appveyor/branch

• ✅ continuous-integration/travis-ci/push.

[testcafe-build-bot]

❌ Tests for the commit 42969f8 have failed. See details:

• ❌ continuous-integration/appveyor/branch

• ✅ continuous-integration/travis-ci/push.

[testcafe-build-bot]

❌ Tests for the commit 0303e8d have failed. See details:

• ✅ continuous-integration/appveyor/branch

• ❌ continuous-integration/travis-ci/push.

[testcafe-build-bot]

❌ Tests for the commit f62d05e have failed. See details:

• ✅ continuous-integration/appveyor/branch

• ❌ continuous-integration/travis-ci/push.

[testcafe-build-bot]

✅ Tests for the commit f62d05e have passed. See details:

• ✅ continuous-integration/appveyor/branch

• ✅ continuous-integration/travis-ci/push.

[testcafe-build-bot]

✅ Tests for the commit 874487a have passed. See details:

• ✅ continuous-integration/appveyor/branch

• ✅ continuous-integration/travis-ci/push.

[AlexanderMoskovkin]

lgtm

[testcafe-build-bot]

✅ Tests for the commit 0565d2d have passed. See details:

• ✅ continuous-integration/appveyor/branch

• ✅ continuous-integration/travis-ci/push.

[helen-dikareva]

lgtm
Let's add in documentation example for skip and only in different style of fixture implementations:

fixture( fixtureName )
fixture `fixtureName`

[VasilyStrelyaev]

Note for tech writers: Current fixture.beforeEach and fixture.afterEach hooks section in documentation is not quite accurate. It's worth mentioning that hooks are run for each test run of the given test. So e.g., if we have 3 browsers set for the test run, each hook will be executed in each browser right before and after the test.

I feel like it can be excessive. Test runs are our internal stuff. We should just say: "Your test needs some initialization code? Put it here!", And it is our job to ensure that it runs in every browser. Users shouldn't bother.