From 25e6e61efc50718650716e06d00c5c34ff5dd8f5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C5=81ukasz=20Niemier?= <lukasz@niemier.pl>
Date: Thu, 17 Sep 2015 13:57:55 +0200
Subject: [PATCH 1/2] First version of RFC

---
 text/0000-machine-readable-tests-output.md | 146 +++++++++++++++++++++
 1 file changed, 146 insertions(+)
 create mode 100644 text/0000-machine-readable-tests-output.md

diff --git a/text/0000-machine-readable-tests-output.md b/text/0000-machine-readable-tests-output.md
new file mode 100644
index 00000000000..701ec7b95f7
--- /dev/null
+++ b/text/0000-machine-readable-tests-output.md
@@ -0,0 +1,146 @@
+- Feature Name: machine\_readable\_tests\_output
+  - Start Date: 2015-09-17
+  - RFC PR: (leave this empty)
+- Rust Issue: (leave this empty)
+
+# Summary
+
+Replace current test output with machine-readable one and add thin compatibility
+layer on top of that.
+
+# Motivation
+
+Currently when testing Rust provide only human-readable output which is nice
+in most cases as we are humans or other humanoids. But from time to time we need
+to feed machine with results of our tests (i.e. some kind of CI that will report
+which tests failed or something like that) and there is no way to do that in
+"civilised" way. We need to parse non-machine-readable output which isn't nice.
+
+# Detailed design
+
+Replace current, human-readable, output with JSON-based output based on
+[TAP-J][tap-j]. For compatibility with existing work flow there should be added
+built-in parser for that protocol which will output in current format.
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
+"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
+interpreted as described in [RFC 2119][rfc2119].
+
+## Protocol description
+
+Output of test MUST be stream of JSON objects that can be parsed by external
+tools.
+
+### Structure
+
+Output is stream of lines containing JSON objects separated by new line. Each
+object MUST contain `type` field which designates `suite`, `test`, `bench` or
+`final`. Any document MAY have `extra` field which contains an open mapping
+for additional, unstandardized fields.
+
+### Suite
+
+  ```json
+{
+  "type": "suite",
+  "name": "Doc-Test",
+  "build": "2015-08-21T10:03:20+0200",
+  "count": 13,
+  "rustc": "2a89bb6ba033b236c79a90486e2e3ee04d0e66f9"
+}
+```
+
+Describes tests suite. It MUST appear only once at the beginning of each stream.
+
+Fields:
+
+| Name    | Description                                                    |
+| ----    | -------------------------------------------------------------- |
+| `type`  | MUST be a `suite`                                              |
+| `build` | MUST be ISO8601 timestamp of build.                            |
+| `name`  | OPTIONAL suite name i.e. "Tests", "Benchmarks" or "Doc-Tests". |
+| `count` | MUST be count of all tests (including ignored).                |
+| `rustc` | MUST be version of Rust compiler used to build test suite.     |
+
+### Test
+
+```json
+{
+  "type": "test",
+  "subtype": "should_panic",
+  "status": "ok",
+  "label": "octavo::digest::md5::tests::test_md5",
+  "file": "src/digest/md5.rs",
+  "line": 684,
+  "stdout": "",
+  "stderr": "",
+  "duration": 100
+}
+```
+
+Describes test. Each test MUST produce only one test object.
+
+Fields:
+
+| Name         | Description                                                                    |
+| ----         | -----------                                                                    |
+| `type`       | MUST be `test`.                                                                |
+| `subtype`    | SHOULD be one of `test`, `bench` or `should_panic`.                            |
+| `status`     | MUST be one of `ok`, `fail` or `ignore`.                                       |
+| `reason`     | MUST describe of failure if `status` is `fail`. Otherwise MUST NOT be present. |
+| `label`      | MUST be unique identifier of test.                                             |
+| `file`       | OPTIONAL file name containing test.                                            |
+| `line`       | OPTIONAL line in `file` that contains test.                                    |
+| `stdout`     | OPTIONAL output of standard output for given test.                             |
+| `stderr`     | OPTIONAL output of standard error for given test.                              |
+| `duration`   | OPTIONAL test execution duration in nanoseconds. MUST be present in benchmark. |
+| `throughput` | OPTIONAL test throughput in case of benchmark test.                            |
+
+### Final
+
+```json
+{
+  "type": "final",
+  "results": {
+    "ok": 10,
+    "fail": 0,
+    "ignore": 2
+  }
+}
+```
+
+Finish test suite and closes stream. Parser MUST ignore all other input unless
+it is new suite.
+
+Fields:
+
+| Name      | Description                                                                                                                                 |
+| ----      | -----------                                                                                                                                 |
+| `type`    | MUST be `final`.                                                                                                                            |
+| `results` | MUST be object containing only 3 fields: `ok`, `fail` and `ignore` which describe how many test passed, failed or was ignored respectfully. |
+
+
+# Drawbacks
+
+This is breaking change in tooling and will require new tool that will provide
+current functionality for compatibility reasons, but IMHO small pain for big gain.
+
+# Alternatives
+
+Do nothing.
+
+# Unresolved questions
+
+Test object fields:
+
+- Should there be any additional fields? How should benchmark test description
+  look like?
+- Maybe there should be additional, optional field `nspi` containing times
+  for each iteration that would help with statistical analysis of benchmarks.
+
+Object separator: is newline enough? Maybe something that will be simpler to parse.
+
+Format: should additional formats be available? Is support only for JSON enough?
+
+[tap-j]: https://github.com/rubyworks/tapout/wiki/TAP-Y-J-Specification
+[rfc2119]: https://www.ietf.org/rfc/rfc2119.txt

From eb752fd1384456f4ff71c1aec5f402fdd38c478c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=C5=81ukasz=20Niemier?= <lukasz@niemier.pl>
Date: Thu, 17 Sep 2015 20:25:46 +0200
Subject: [PATCH 2/2] Minor fixes

---
 text/0000-machine-readable-tests-output.md | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/text/0000-machine-readable-tests-output.md b/text/0000-machine-readable-tests-output.md
index 701ec7b95f7..83d71c25752 100644
--- a/text/0000-machine-readable-tests-output.md
+++ b/text/0000-machine-readable-tests-output.md
@@ -5,8 +5,7 @@
 
 # Summary
 
-Replace current test output with machine-readable one and add thin compatibility
-layer on top of that.
+Replace current test output with machine-readable one.
 
 # Motivation
 
@@ -85,7 +84,7 @@ Fields:
 | Name         | Description                                                                    |
 | ----         | -----------                                                                    |
 | `type`       | MUST be `test`.                                                                |
-| `subtype`    | SHOULD be one of `test`, `bench` or `should_panic`.                            |
+| `subtype`    | SHOULD be one of `test`, `bench` or `should_panic`. Defaults to `test`.        |
 | `status`     | MUST be one of `ok`, `fail` or `ignore`.                                       |
 | `reason`     | MUST describe of failure if `status` is `fail`. Otherwise MUST NOT be present. |
 | `label`      | MUST be unique identifier of test.                                             |