Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(testing): Add behavior-driven development #2067

Merged
merged 20 commits into from
Apr 8, 2022
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
304 changes: 302 additions & 2 deletions testing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,306 @@ Clears all registered benchmarks, so calling `runBenchmarks()` after it wont run
them. Filtering can be applied by setting `BenchmarkRunOptions.only` and/or
`BenchmarkRunOptions.skip` to regular expressions matching benchmark names.

## Behavior-driven development

With the `bdd.ts` module you can write your tests in a familiar format for
grouping tests and adding setup/teardown hooks used by other JavaScript testing
frameworks like Jasmine, Jest, and Mocha.

The `describe` function creates a block that groups together several related
tests. The `it` function registers an individual test case. The `describe` and
`it` functions have similar call signatures to `Deno.test`, making it easy to
migrate from using `Deno.test`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's drop the last clause as we generally don't recommend migration from Deno.test to BDD (more descriptions are in the below comment)

Suggested change
`it` functions have similar call signatures to `Deno.test`, making it easy to
migrate from using `Deno.test`.
`it` functions have similar call signatures to `Deno.test`.


### Hooks

There are 4 types of hooks available for test suites. A test suite can have
multiples of each type of hook, they will be called in the order that they are
registered. The `afterEach` and `afterAll` hooks will be called whether or not
the test case passes. The all hooks will be called once for the whole group
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should write 'All' hooks instead of all hooks to make it clear what are referenced

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*All hooks

while the each hooks will be called for each individual test case.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*Each hooks


- `beforeAll`: Runs before all of the tests in the test suite.
- `afterAll`: Runs after all of the tests in the test suite finish.
- `beforeEach`: Runs before each of the individual test cases in the test suite.
- `afterEach`: Runs after each of the individual test cases in the test suite.

If a hook is registered at the top level, a global test suite will be registered
and all tests will belong to it. Hooks registered at the top level must be
registered before any individual test cases or test suites.

### Focusing tests

If you would like to only run specific individual test cases, you can do so by
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you would like to only run specific individual test cases, you can do so by
If you would like to run only specific test cases, you can do so by

calling `it.only` instead of `it`. If you would like to only run specific test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
calling `it.only` instead of `it`. If you would like to only run specific test
calling `it.only` instead of `it`. If you would like to run only specific test

suites, you can do so by calling `describe.only` instead of `describe`.

There is one limitation to this when the individual test cases or test suites
belong to another test suite, they will be the only ones to run within the top
level test suite.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you provide an example here? It's not obviously clear what this sentence means.

Copy link
Contributor Author

@KyleJune KyleJune Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you look at one of the user test examples I have, this issue can be reproduced by adding another test case or describe block that doesn't belong to the User describe block. If you add .only to the test for getAge that will be the only test case to run within the top level User describe block. The other test cases that do not belong to the User describe block will also still run. currently the only workaround to ensure the user test step is the only one in the file to run is also adding .only to the User describe block so that when Deno.test is called, it will be called with the only flag.

I'll still add an example to readme for this limitation. Just thought I'd try explaining it more here first.

The reason this problem exists is because Deno.test is called the moment describe is called, and at that time, it doesn't know if any of it's steps have the only flag. If I had a hook into when tests are about to run, I could delay registration until the last moment, allowing me to determine if Deno.test should be called with only flag based on if one of it's child steps has the only flag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I thought about this more and think I could resolve limitation for test cases registered within the describe callback. It's just the flat style where it wouldn't be known at time of Deno.test call if a step registered after the describe call has the only flag.


### Ignoring tests

If you would like to not run specific individual test cases, you can do so by
calling `it.ignore` instead of `it`. If you would like to only run specific test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
calling `it.ignore` instead of `it`. If you would like to only run specific test
calling `it.ignore` instead of `it`. If you would like to not run specific test

suites, you can do so by calling `describe.ignore` instead of `describe`.

### Sanitization options

Like `Deno.TestDefinition`, the `DescribeDefinition` and `ItDefinition` have
sanitization options. They work in the same way.

- sanitizeExit: Ensure the test case does not prematurely cause the process to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- sanitizeExit: Ensure the test case does not prematurely cause the process to
- `sanitizeExit`: Ensure the test case does not prematurely cause the process to

exit, for example via a call to Deno.exit. Defaults to true.
- sanitizeOps: Check that the number of async completed ops after the test is
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- sanitizeOps: Check that the number of async completed ops after the test is
- `sanitizeOps`: Check that the number of async completed ops after the test is

the same as number of dispatched ops. Defaults to true.
- sanitizeResources: Ensure the test case does not "leak" resources - ie. the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- sanitizeResources: Ensure the test case does not "leak" resources - ie. the
- `sanitizeResources`: Ensure the test case does not "leak" resources - ie. the

resource table after the test has exactly the same contents as before the
test. Defaults to true.

### Permissions option

Like `Deno.TestDefinition`, the `DescribeDefintion` and `ItDefinition` have a
permissions option. They specify the permissions that should be used to run an
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
permissions option. They specify the permissions that should be used to run an
`permissions` option. They specify the permissions that should be used to run an

individual test case or test suite. Set this to "inherit" to keep the calling
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
individual test case or test suite. Set this to "inherit" to keep the calling
individual test case or test suite. Set this to `"inherit"` to keep the calling

thread's permissions. Set this to "none" to revoke all permissions.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
thread's permissions. Set this to "none" to revoke all permissions.
thread's permissions. Set this to `"none"` to revoke all permissions.


Defaults to "inherit".
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Defaults to "inherit".
This setting defaults to `"inherit"`.


There is currently one limitation to this, you cannot use the permissions option
on an individual test case or test suite that belongs to another test suite.

### Migrating and usage
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comparing describe / it APIs with Deno.test is very good, but we generally don't recommend migrating from Deno.test suites to BDD style testing (Deno.test should still be the default way of writing tests and BDD style should be an optional convenient way of writing them if the users prefer) So the section name here sounds concerning to me.

Could you rephrase the section name to something like Comparing to Deno.test?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with that change. I just worded it that way as a way to describe how someone could switch to using bdd.ts if they prefer this style over the default style that they may already be using at the time of finding this. I'll try rewriting this in a way that makes it clear this isn't the preferred way of writing tests for Deno and that it's just another way of writing them.


To migrate from `Deno.test`, all you have to do is replace `Deno.test` with
`it`. If you are using the step API, you will need to replace `Deno.test` with
describe and steps with `describe` or `it`. The callback for individual test
cases can be syncronous or asyncronous.

Below is an example of a test file using `Deno.test` and `t.step`. In the
following sections there are examples of how it can be converted to using nested
test grouping, flat test grouping, and a mix of both.

```ts
// https://deno.land/std@$STD_VERSION/testing/bdd_examples/user_test.ts
import {
assertEquals,
assertStrictEquals,
assertThrows,
} from "https://deno.land/std@$STD_VERSION/testing/asserts.ts";
import { User } from "https://deno.land/std@$STD_VERSION/testing/bdd_examples/user.ts";

Deno.test("User.users initially empty", () => {
assertEquals(User.users.size, 0);
});

Deno.test("User constructor", () => {
try {
const user = new User("Kyle");
assertEquals(user.name, "Kyle");
assertStrictEquals(User.users.get("Kyle"), user);
} finally {
User.users.clear();
}
});

Deno.test("User age", async (t) => {
const user = new User("Kyle");

await t.step("getAge", () => {
assertThrows(() => user.getAge(), Error, "Age unknown");
user.age = 18;
assertEquals(user.getAge(), 18);
});

await t.step("setAge", () => {
user.setAge(18);
assertEquals(user.getAge(), 18);
});
});
```

#### Nested test grouping

Tests created within the callback of a `describe` function call will belong to
the new test suite it creates. The hooks can be created within it or be added to
the options argument for describe.

```ts
// https://deno.land/std@$STD_VERSION/testing/bdd_examples/user_nested_test.ts
import {
assertEquals,
assertStrictEquals,
assertThrows,
} from "https://deno.land/std@$STD_VERSION/testing/asserts.ts";
import {
afterEach,
beforeEach,
describe,
it,
} from "https://deno.land/std@$STD_VERSION/testing/bdd.ts";
import { User } from "https://deno.land/std@$STD_VERSION/testing/bdd_examples/user.ts";

describe("User", () => {
it("users initially empty", () => {
assertEquals(User.users.size, 0);
});

it("constructor", () => {
try {
const user = new User("Kyle");
assertEquals(user.name, "Kyle");
assertStrictEquals(User.users.get("Kyle"), user);
} finally {
User.users.clear();
}
});

describe("age", () => {
let user: User;

beforeEach(() => {
user = new User("Kyle");
});

afterEach(() => {
User.users.clear();
});

it("getAge", function () {
assertThrows(() => user.getAge(), Error, "Age unknown");
user.age = 18;
assertEquals(user.getAge(), 18);
});

it("setAge", function () {
user.setAge(18);
assertEquals(user.getAge(), 18);
});
});
});
```

#### Flat test grouping

The `describe` function returns a unique symbol that can be used to reference
the test suite for adding tests to it without having to create them within a
callback. The gives you the ability to have test grouping without any extra
indentation in front of the grouped tests.

```ts
// https://deno.land/std@$STD_VERSION/testing/bdd_examples/user_flat_test.ts
import {
assertEquals,
assertStrictEquals,
assertThrows,
} from "https://deno.land/std@$STD_VERSION/testing/asserts.ts";
import {
describe,
it,
} from "https://deno.land/std@$STD_VERSION/testing/bdd.ts";
import { User } from "https://deno.land/std@$STD_VERSION/testing/bdd_examples/user.ts";

const userTests = describe("User");

it(userTests, "users initially empty", () => {
assertEquals(User.users.size, 0);
});

it(userTests, "constructor", () => {
try {
const user = new User("Kyle");
assertEquals(user.name, "Kyle");
assertStrictEquals(User.users.get("Kyle"), user);
} finally {
User.users.clear();
}
});

const ageTests = describe({
name: "age",
suite: userTests,
beforeEach(this: { user: User }) {
this.user = new User("Kyle");
},
afterEach() {
User.users.clear();
},
});

it(ageTests, "getAge", function () {
const { user } = this;
assertThrows(() => user.getAge(), Error, "Age unknown");
user.age = 18;
assertEquals(user.getAge(), 18);
});

it(ageTests, "setAge", function () {
const { user } = this;
user.setAge(18);
assertEquals(user.getAge(), 18);
});
```

#### Mixed test grouping

Both nested test grouping and flat test grouping can be used together. This can
be useful if you'd like to create deep groupings without all the extra
indentation in front of each line.

```ts
// https://deno.land/std@$STD_VERSION/testing/bdd_examples/user_mixed_test.ts
import {
assertEquals,
assertStrictEquals,
assertThrows,
} from "https://deno.land/std@$STD_VERSION/testing/asserts.ts";
import {
describe,
it,
} from "https://deno.land/std@$STD_VERSION/testing/bdd.ts";
import { User } from "https://deno.land/std@$STD_VERSION/testing/bdd_examples/user.ts";

describe("User", () => {
it("users initially empty", () => {
assertEquals(User.users.size, 0);
});

it("constructor", () => {
try {
const user = new User("Kyle");
assertEquals(user.name, "Kyle");
assertStrictEquals(User.users.get("Kyle"), user);
} finally {
User.users.clear();
}
});

const ageTests = describe({
name: "age",
beforeEach(this: { user: User }) {
this.user = new User("Kyle");
},
afterEach() {
User.users.clear();
},
});

it(ageTests, "getAge", function () {
const { user } = this;
assertThrows(() => user.getAge(), Error, "Age unknown");
user.age = 18;
assertEquals(user.getAge(), 18);
});

it(ageTests, "setAge", function () {
const { user } = this;
user.setAge(18);
assertEquals(user.getAge(), 18);
});
});
```

## Mocking

Test spies are function stand-ins that are used to assert if a function's
Expand Down Expand Up @@ -412,8 +712,8 @@ second example.

Say we have two functions, `randomMultiple` and `randomInt`, if we want to
assert that `randomInt` is called during execution of `randomMultiple` we need a
way to spy on the `randomInt` function. That could be done with either either of
the spying techniques previously mentioned. To be able to verify that the
way to spy on the `randomInt` function. That could be done with either of the
spying techniques previously mentioned. To be able to verify that the
`randomMultiple` function returns the value we expect it to for what `randomInt`
returns, the easiest way would be to replace the `randomInt` function's behavior
with more predictable behavior.
Expand Down
Loading