Add predicate capturing results of test runs

Signed-off-by: Aditya Sirish <[email protected]>

spec/predicates/test-results.md

@@ -0,0 +1,163 @@
+# Predicate type: Test Results
+
+Type URI: https://in-toto.io/attestation/test-result/v0.1
+
+Version: 0.1.0
+
+Authors: Aditya Sirish A Yelgundhalli (@adityasaky)
+
+## Purpose
+
+This predicate type defines a generic schema to express the result of running
+tests in software supply chains. The schema may be extended to support different
+types of testing or specific test harnesses.
+
+## Use Cases
+
+Software development processes include several types of tests. This attestation
+can be used to express the results of running those tests. It can be used to
+verify:
+
+1.  that all tests were in fact run, and
+2.  that all required tests passed
+
+Therefore, each attestation corresponds to one invocation of a test suite, and
+may include the results of several individual tests.
+
+### Asserting Test Configurations Used
+
+The supply chain owner creates a policy that records the expected test
+configurations. During verification, the policy checks that the test attestation
+used the right configurations. If verified using in-toto layouts, a custom
+inspection may optionally parse the `testRun.link` field to verify attestation
+matches the test run.
+
+### Asserting Test Results
+
+In addition to the previous use case, the supply chain owner creates a policy
+verifying that test results passed. In the simplest case, the policy applies to
+all tests. Therefore, it asserts the contents of `result` and that
+`testRun.failedTests` is empty. In more nuanced cases, a subset of tests may
+matter. For example, the tested artifact may be an OS image that's to be
+deployed to three types of devices. As such, the test harness tests the new
+image on an instance of each device. When verifying the attestation prior to an
+image being installed on a device, it only matters that the tests passed on the
+corresponding test device and not necessarily the others.
+
+As before, if the attestation is verified using an in-toto layout, a custom
+inspection may examine the `testRun.link` contents to verify the contents of the
+attestation.
+
+## Prerequisites
+
+Understanding of the
+[in-toto attestation specification](https://github.com/in-toto/attestation).
+
+## Model
+
+This predicate type includes two compulsory fields, `result`, that describes the
+result of the test run, and `testRun` object contains the configuration of the
+tests, as well as optionally a link to the test run, and lists of tests that
+passed, failed, and passed with warnings.  The expected `subject` for this are
+the source artifacts themselves or some reference to them such as a Git commit.
+
+## Schema
+
+```json
+{
+    "_type": "https://in-toto.io/Statement/v1",
+    "subject": [{...}],
+    "predicateType": "https://in-toto.io/attestation/test-result/v0.1",
+    "predicate": {
+        "result": "pass|fail",
+        "testRun": {
+            "link": "<URL>",
+            "configuration": ["<ResourceDescriptor>", ...],
+            "passedTests": ["<TEST_NAME>", ...],
+            "warnedTests": ["<TEST_NAME>", ...],
+            "failedTests": ["<TEST_NAME>", ...]
+        }
+    }
+}
+```
+
+### Parsing Rules
+
+This predicate follows the
+[in-toto Attestation Framework's parsing rules](../v1.0/README.md#parsing-rules).
+
+### Fields
+
+`result` _boolean_ , _required_
+
+Indicates the result of the test run. If true, it indicates _all_ tests passed
+in the corresponding run.
+
+`testRun` _object_, _required_
+
+`testRun.link` _URI_, _optional_
+
+Contains a link to the test run. Useful to point to information that cannot be
+captured in the attestation.
+
+`testRun.configuration` _list of ResourceDescriptor_, _required_
+
+Reference to the configuration used for the test run.
+
+`testRun.passedTests` _list_, _optional_
+
+Each entry corresponds to a single test that passed.
+
+`testRun.warnedTests` _list_, _optional_
+
+Each entry corresponds to a single test that expressed a warning.
+
+`testRun.failedTests` _list_, _optional_
+
+Each entry corresponds to a single test that failed.
+
+## Example
+
+```json
+{
+    "_type": "https://in-toto.io/Statement/v1",
+    "subject": [
+        {
+            "digest": {
+                "gitCommit": "d20ace7968ba43c0219f62d71334c1095bab1602"
+            }
+        }
+    ],
+    "predicateType": "https://in-toto.io/attestation/test-result/v0.1",
+    "predicate": {
+        "result": "pass",
+        "testRun": {
+            "link": "https://github.com/in-toto/in-toto/actions/runs/4425592351",
+            "configuration": [{
+                "name": ".github/workflows/ci.yml",
+                "downloadLocation": "https://github.com/in-toto/in-toto/blob/d20ace7968ba43c0219f62d71334c1095bab1602/.github/workflows/ci.yml",
+                "digest": {
+                    "gitBlob": "ebe4add40f63c3c98bc9b32ff1e736f04120b023"
+                }
+            }],
+            "passedTests": [
+                "build (3.7, ubuntu-latest, py)",
+                "build (3.7, macos-latest, py)",
+                "build (3.7, windows-latest, py)",
+                "build (3.8, ubuntu-latest, py)",
+                "build (3.8, macos-latest, py)",
+                "build (3.8, windows-latest, py)",
+                "build (3.9, ubuntu-latest, py)",
+                "build (3.9, macos-latest, py)",
+                "build (3.9, windows-latest, py)",
+                "build (3.10, ubuntu-latest, py)",
+                "build (3.10, macos-latest, py)",
+                "build (3.10, windows-latest, py)",
+                "build (3.x, ubuntu-latest, lint)"
+            ],
+            "warnedTests": [],
+            "failedTests": []
+        }
+    }
+}
+```