Machine-readable output of tests #1284
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| - Feature Name: machine\_readable\_tests\_output | ||
| - Start Date: 2015-09-17 | ||
| - RFC PR: (leave this empty) | ||
| - Rust Issue: (leave this empty) | ||
|
|
||
| # Summary | ||
|
|
||
| Replace current test output with machine-readable one. | ||
|
|
||
| # Motivation | ||
|
|
||
| Currently when testing Rust provide only human-readable output which is nice | ||
| in most cases as we are humans or other humanoids. But from time to time we need | ||
| to feed machine with results of our tests (i.e. some kind of CI that will report | ||
| which tests failed or something like that) and there is no way to do that in | ||
| "civilised" way. We need to parse non-machine-readable output which isn't nice. | ||
|
|
||
| # Detailed design | ||
|
|
||
| Replace current, human-readable, output with JSON-based output based on | ||
| [TAP-J][tap-j]. For compatibility with existing work flow there should be added | ||
| built-in parser for that protocol which will output in current format. | ||
|
|
||
| The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", | ||
| "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be | ||
| interpreted as described in [RFC 2119][rfc2119]. | ||
|
|
||
| ## Protocol description | ||
|
|
||
| Output of test MUST be stream of JSON objects that can be parsed by external | ||
| tools. | ||
|
|
||
| ### Structure | ||
|
|
||
| Output is stream of lines containing JSON objects separated by new line. Each | ||
| object MUST contain `type` field which designates `suite`, `test`, `bench` or | ||
| `final`. Any document MAY have `extra` field which contains an open mapping | ||
| for additional, unstandardized fields. | ||
|
|
||
| ### Suite | ||
|
|
||
| ```json | ||
| { | ||
| "type": "suite", | ||
| "name": "Doc-Test", | ||
| "build": "2015-08-21T10:03:20+0200", | ||
| "count": 13, | ||
| "rustc": "2a89bb6ba033b236c79a90486e2e3ee04d0e66f9" | ||
| } | ||
| ``` | ||
|
|
||
| Describes tests suite. It MUST appear only once at the beginning of each stream. | ||
|
|
||
| Fields: | ||
|
|
||
| | Name | Description | | ||
| | ---- | -------------------------------------------------------------- | | ||
| | `type` | MUST be a `suite` | | ||
| | `build` | MUST be ISO8601 timestamp of build. | | ||
| | `name` | OPTIONAL suite name i.e. "Tests", "Benchmarks" or "Doc-Tests". | | ||
| | `count` | MUST be count of all tests (including ignored). | | ||
| | `rustc` | MUST be version of Rust compiler used to build test suite. | | ||
|
|
||
| ### Test | ||
|
|
||
| ```json | ||
| { | ||
| "type": "test", | ||
| "subtype": "should_panic", | ||
| "status": "ok", | ||
| "label": "octavo::digest::md5::tests::test_md5", | ||
| "file": "src/digest/md5.rs", | ||
| "line": 684, | ||
| "stdout": "", | ||
| "stderr": "", | ||
| "duration": 100 | ||
| } | ||
| ``` | ||
|
|
||
| Describes test. Each test MUST produce only one test object. | ||
|
|
||
| Fields: | ||
|
|
||
| | Name | Description | | ||
| | ---- | ----------- | | ||
| | `type` | MUST be `test`. | | ||
| | `subtype` | SHOULD be one of `test`, `bench` or `should_panic`. Defaults to `test`. | | ||
| | `status` | MUST be one of `ok`, `fail` or `ignore`. | | ||
| | `reason` | MUST describe of failure if `status` is `fail`. Otherwise MUST NOT be present. | | ||
| | `label` | MUST be unique identifier of test. | | ||
| | `file` | OPTIONAL file name containing test. | | ||
| | `line` | OPTIONAL line in `file` that contains test. | | ||
| | `stdout` | OPTIONAL output of standard output for given test. | | ||
| | `stderr` | OPTIONAL output of standard error for given test. | | ||
| | `duration` | OPTIONAL test execution duration in nanoseconds. MUST be present in benchmark. | | ||
| | `throughput` | OPTIONAL test throughput in case of benchmark test. | | ||
|
|
||
| ### Final | ||
|
|
||
| ```json | ||
| { | ||
| "type": "final", | ||
| "results": { | ||
| "ok": 10, | ||
| "fail": 0, | ||
| "ignore": 2 | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Finish test suite and closes stream. Parser MUST ignore all other input unless | ||
| it is new suite. | ||
|
|
||
| Fields: | ||
|
|
||
| | Name | Description | | ||
| | ---- | ----------- | | ||
| | `type` | MUST be `final`. | | ||
| | `results` | MUST be object containing only 3 fields: `ok`, `fail` and `ignore` which describe how many test passed, failed or was ignored respectfully. | | ||
|
|
||
|
|
||
| # Drawbacks | ||
|
|
||
| This is breaking change in tooling and will require new tool that will provide | ||
| current functionality for compatibility reasons, but IMHO small pain for big gain. | ||
|
There was a problem hiding this comment. Is the test output a committed API? Using the test API directly requires #[feature(test)], so it is not breaking a committed API. There was a problem hiding this comment. No, test isn't commited. But testing is available in stable Rust and change in that matter will be visible to non-compiler-devs also so I marked it as a breaking change. There was a problem hiding this comment. Why not make the machine-readable output an option and keep the human-readable output the default? Machine-readable output by default makes for a dreadful interactive experience. FWIW pytest outputs a human-readable format to stdout and optionally a machine-readable one to a specified file. The ability to output both human-readable and machine-readable is actually convenient in CI, you can send the human-readable output to a human-readable log and the machine-readable output to the CI's parser. There was a problem hiding this comment. That's the point of drawbacks. That this should be written, but IMHO isn't point of this RFC. I can rewrite it to provide flag that will print out machine-readable style. About writing to file machine-readable and on stdout human-readable then I think that one should use Unix tools: Where |
||
|
|
||
| # Alternatives | ||
|
|
||
| Do nothing. | ||
|
|
||
| # Unresolved questions | ||
|
|
||
| Test object fields: | ||
|
|
||
| - Should there be any additional fields? How should benchmark test description | ||
| look like? | ||
| - Maybe there should be additional, optional field `nspi` containing times | ||
| for each iteration that would help with statistical analysis of benchmarks. | ||
|
|
||
| Object separator: is newline enough? Maybe something that will be simpler to parse. | ||
|
|
||
| Format: should additional formats be available? Is support only for JSON enough? | ||
|
|
||
| [tap-j]: https://github.com/rubyworks/tapout/wiki/TAP-Y-J-Specification | ||
| [rfc2119]: https://www.ietf.org/rfc/rfc2119.txt | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would make sense to agree on something like http://dataprotocols.org/ndjson/ or http://jsonlines.org/ for that or is that up to TAP-J to define?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It could also make usage of
\1EASCII code which refers to "Record Separator". This is still under consideration, but 1 record per line is simple enough.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd take '\n' or '\r\n', it's a pretty common format in the NoSQL-world (e.g. Elasticsearch bulk uses it) and JSON can be written newline-free very well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But is inconsistent between platform as Windows still uses
\r\nas a line separator and I believe thatprintln!macro respect that. Although this needs reconsideration as I chosen new line as it was simple enough to implement in first draft of specs.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That inconsistency doesn't matter in that case as '{"test": "foo"}\r' is the same JSON value as '{"test": "foo"}'. Even if your IO isn't prepared for that, this is not an issue.