-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs: add introduction and guide to readme #13
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,298 @@ | ||
# scenario-tester | ||
|
||
A tool for managing many combinations of real and fake NPM packages within your test suite. | ||
A tool for managing many combinations of real and fake npm packages within your test suite. | ||
|
||
Extracted from [ember-auto-import](https://github.com/ef4/ember-auto-import). No docs here yet, but: | ||
- you can see how it's used in ember-auto-import/test-scenarios | ||
- it's all typescript so you can follow the types to learn what's possible | ||
- it's all typescript so you can follow the types to learn what's possible | ||
|
||
## Problem to solve | ||
|
||
In the context of `scenario-tester`, a *project* is a codebase on your filesystem that you want to test against. | ||
|
||
Specifically, you may want to test an app against various combinations of npm packages. Or maybe you're working on an npm package and you want to ensure that different versions of your pacakge work correctly with different consuming apps representing different use cases. In these examples, the codebases of both test apps and packages are called "projects". | ||
|
||
To test all combinations of apps and packages, you'd need to have them on disk. Those test codebases will be mostly identical, varying only in which dependencies (or even versions of the same dependency) are installed. | ||
|
||
Maintaining such a collection of test project codebases can be a burden. For example, if you need to upgrade a third-party dependency, you will have to repeat the same steps in every test project codebase. There's also a risk of your projects getting out of sync. | ||
|
||
## How scenario-tester solves the problem | ||
|
||
With `scenario-tester`, you only need to maintain a small number of project templates. In the simplest case, it will be just one test app or package. For complex cases, several template projects may be used. | ||
|
||
`scenario-tester` lets you take a template project and customize it by creating *scenarios*: copies of the project containing unique changes. For example, each scenario could have a different version of an npm dependency added to the project. And then you can use your JS test suite of choice to run tests against each scenario. | ||
|
||
## Example usage | ||
|
||
Here's a very basic example: | ||
|
||
```js | ||
import { Scenarios } from 'scenario-tester'; | ||
import { Project } from 'fixturify-project'; | ||
import { module, test } from 'qunit'; | ||
|
||
Scenarios | ||
|
||
// Use a project from filesystem as a template | ||
.fromProject(() => Project.fromDir(dirname(require.resolve('../test-app/package.json')), { linkDevDeps: true }))) | ||
|
||
// Define multiple scenarios, each scenario is derived from the template and has unique dependencies installed | ||
.expand({ | ||
modern: async (project) => { | ||
project.linkDevDependency('some-npm-package', { baseDir: __dirname, resolveName: 'some-npm-package' }); | ||
project.linkDevDependency('another-npm-package', { baseDir: __dirname, resolveName: 'another-npm-package' }); | ||
}, | ||
|
||
legacy: async function (project) => { | ||
project.linkDevDependency('some-npm-package', { baseDir: __dirname, resolveName: 'some-npm-package--legacy' }); | ||
project.linkDevDependency('another-npm-package', { baseDir: __dirname, resolveName: 'another-npm-package--legacy' }); | ||
} | ||
}) | ||
|
||
// Emit derivative projects to the filesystem | ||
.forEachScenario(scenario => { | ||
|
||
// Define your test suite | ||
module(scenario.name, function (hooks) { | ||
hooks.beforeEach(assert => { | ||
// ... | ||
}); | ||
|
||
test('output directories exist', async function () { | ||
// ... | ||
}); | ||
|
||
test('package.json is modified appropriately', async function () { | ||
/// ... | ||
}); | ||
}); | ||
}); | ||
``` | ||
|
||
Note: no new project codebases will be emitted to your filesystem until `forEachScenario` is executed. This lets you manipulate scenarios before emitting their codebases. | ||
|
||
## Creating and manipulating project codebases | ||
|
||
Instead of defining a template project from disk as shown above, you can do it programmatically. | ||
|
||
`scenario-tester` relies on [stefanpenner/node-fixturify-project](https://github.com/stefanpenner/node-fixturify-project) (npm package [fixturify-project](https://www.npmjs.com/package/fixturify-project)). | ||
|
||
`fixturify-project` converts codebases between a filesystem representation and a JSON representation. For example, it lets you define a project (a test app or an npm package) as JSON: | ||
|
||
```js | ||
const { Project } = require('fixturify-project'); | ||
|
||
const project = new Project('rsvp', '3.1.4', { | ||
files: { | ||
'filename1.js': 'module.exports = "Hello, World!"', | ||
dirname1: { | ||
'filename2.json': `{ | ||
"This is a": "multiline string", | ||
"that represents": "a JSON file in your project", | ||
}`, | ||
}, | ||
}, | ||
}); | ||
``` | ||
|
||
Regardless of whether you created a project from disk or JSON, it will be represented in a `Project` class instance from `fixturify-project`. | ||
|
||
You can emit your project codebase to filesystem: | ||
|
||
```js | ||
project.basedir = '/tmp/my-project'; // Optional. When not set, a random tmp path will be assigned to `basedir`. | ||
|
||
await project.write() | ||
``` | ||
|
||
This produces the following structure on disk: | ||
|
||
```sh | ||
/tmp/my-project/filename1.js | ||
/tmp/my-project/dirname1/filename2.js | ||
``` | ||
|
||
You can add dependencies to a project: | ||
|
||
```js | ||
project.addDependency('mocha', '5.2.0'); | ||
project.addDependency('chai', '5.2.0'); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. if you want to show addDependency here use fake/mock package names. These dependencies that are added have nothing to do with npm, they are created as fixturify-project Projects with a minimal setup to be considered an npm package and then added to your project see https://github.com/embroider-build/embroider/blob/main/tests/scenarios/compat-addon-import-test.ts#L10-L32 for inspiration There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Changed. |
||
|
||
await project.write(); | ||
``` | ||
|
||
This will result in the follwing file/folder structure: | ||
|
||
```sh | ||
/tmp/my-project/package.json | ||
|
||
/tmp/my-project/filename1.js | ||
/tmp/my-project/dirname1/filename2.json | ||
|
||
# Dummy node_modules for added dependencies | ||
/tmp/my-project/node_modules/mocha/index.js | ||
/tmp/my-project/node_modules/mocha/package.json | ||
/tmp/my-project/node_modules/chai/index.json | ||
/tmp/my-project/node_modules/chai/package.json | ||
``` | ||
|
||
❓ You can then replace dummy `node_modules/` with real ones: | ||
|
||
``` | ||
cd /tmp/my-project | ||
rm -r node_modules/ | ||
pnpm i | ||
``` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. delete this section 👍 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. |
||
|
||
## Providing multiple versions of the same package | ||
|
||
In the example above, two different versions of the same package are included into scenarios: | ||
|
||
```js | ||
project.linkDevDependency('some-npm-package', { baseDir: __dirname, resolveName: 'some-npm-package' }); | ||
project.linkDevDependency('some-npm-package', { baseDir: __dirname, resolveName: 'some-npm-package--legacy' }); | ||
``` | ||
|
||
Normally, you don't have different versions of the same npm package published under different package names, whereas package resolution built into Node does not let you require a specific version. | ||
|
||
You can work around this issue. | ||
|
||
First, create a `package.json` file next to the JS file where you define scenarios. This implies that they are in a subfolder: | ||
|
||
```sh | ||
package.json | ||
|
||
tests/scenarios.js | ||
tests/package.json | ||
``` | ||
|
||
In the new `package.json`, you can include multiple versions of the same package, exposing them into your app under different package names. | ||
|
||
```json | ||
{ | ||
"name": "scenarios", | ||
"version": "0.0.1", | ||
"description": "", | ||
"devDependencies": { | ||
"some-npm-package": "npm:some-npm-package@latest", | ||
"some-npm-package--legacy": "npm:[email protected]", | ||
}, | ||
} | ||
``` | ||
|
||
Now you can reference them in your app as if they are different packages. | ||
|
||
## Reusing and customizing scenarios | ||
|
||
You don't need this section if you define scenarios and run tests in the same file. | ||
|
||
But if your test suite is complex, you may want to define scenarios in a separate file, import them into multiple test files. This lets you pick only a subset of scenarios for each test file and to customize them further. | ||
|
||
You can achieve that with the following methods on the `Scenarios` class: | ||
|
||
* `skip` — removes one scenario with the given name from collection, returns new collection; | ||
* `only` — returns new collection containing only the scenario with the given name; | ||
* `filter` (⚠ does not exist yet) — returns a subset of scenarios, filtered using a callback; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. delete this line 👍 let's not document things that don't exist There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Removed. |
||
* `map` — lets you all scenarios in collection. | ||
|
||
Consider this example: | ||
|
||
```js | ||
// tests/scenarios.js | ||
import { Scenarios } from 'scenario-tester'; | ||
import { Project } from 'fixturify-project'; | ||
|
||
export default Scenarios | ||
|
||
// Use a template project from filesystem | ||
.fromProject(() => Project.fromDir(dirname(require.resolve('../test-app/package.json')), { linkDevDeps: true }))) | ||
|
||
// Define multiple scenarios, each scenario is derived from the template and has unique dependencies installed | ||
.expand({ | ||
modern: async (project) => { | ||
project.linkDevDependency('some-npm-package', { baseDir: __dirname, resolveName: 'some-npm-package' }); | ||
project.linkDevDependency('another-npm-package', { baseDir: __dirname, resolveName: 'another-npm-package' }); | ||
}, | ||
|
||
legacy: async function (project) => { | ||
project.linkDevDependency('some-npm-package', { baseDir: __dirname, resolveName: 'some-npm-package--legacy' }); | ||
project.linkDevDependency('another-npm-package', { baseDir: __dirname, resolveName: 'another-npm-package--legacy' }); | ||
} | ||
}) | ||
``` | ||
|
||
```js | ||
// tests/legacy-test.js | ||
import scenarios from './scenarios'; | ||
import { module, test } from 'qunit'; | ||
|
||
scenarios | ||
// Pick one scenario | ||
.only('legacy') | ||
|
||
// Customize it | ||
.map('without-another-npm-package', project => { // This callback can also be async | ||
project.removeDevDependency('another-npm-package--legacy'); | ||
}) | ||
|
||
// Emit derivative projects to the filesystem | ||
.forEachScenario(scenario => { | ||
|
||
// Define your test suite | ||
module(scenario.name, function (hooks) { | ||
hooks.beforeEach(assert => { | ||
// ... | ||
}); | ||
|
||
test('output directories exist', async function () { | ||
// ... | ||
}); | ||
|
||
test('package.json is modified appropriately', async function () { | ||
/// ... | ||
}); | ||
}); | ||
}); | ||
``` | ||
|
||
## Command-line interface | ||
|
||
It is recommended that you define the following three commands in your `package.json` (the one specific to your scenarios). | ||
|
||
Example, customize to your needs: | ||
|
||
```json | ||
{ | ||
"scripts": { | ||
"test": "qunit *-test.ts", | ||
"test:list": "scenario-tester list", | ||
"test:output": "scenario-tester output" | ||
} | ||
} | ||
``` | ||
|
||
The `test` command should be specific to your test suite of choice. | ||
|
||
The other two commands delegate to `scenario-tester`: | ||
|
||
* `scenario-tester list` — enumerates defined scenarios. | ||
|
||
This command will output a list of all scenario names defined in your test suite. | ||
|
||
Arguments: | ||
|
||
* `--require <package-name>` — require a package before starting, e. g. `ts-node/register` if your test files are in TypeScript | ||
* `--files <glob>` — which files to look in. | ||
* `--matrix <format string>` — ❓ process scenario names with [util.format](https://nodejs.org/api/util.html#utilformatformat-args) and output as JSON. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe add a section about CI setup to the guide? Maybe take from https://github.com/ef4/scenario-tester/pull/10/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R84 essentially There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have mentioned that this flag is used in Github CI test matrix config, but have not provided full CI documentation. I suggest that it goes into |
||
|
||
* `scenario-tester output` — emit derivative projects to filesystem without running the test suite. | ||
|
||
Useful for debugging. | ||
|
||
Arguments: | ||
|
||
* `--require <package-name>` — require a package before starting, e. g. `ts-node/register` if your test files are in TypeScript | ||
* `--files <glob>` — which files to look in. | ||
* `--scenario <name>` — pick only one specific scenario. | ||
* `--outdir <path>` — path to directory, where to emit a derivative project codebase of the scenario. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can link to one or the other github or npm? they link back to each other so it doesn't matter which one we link to here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.