Exercism Test Runner Template

This repository is a template repository for creating test runners for Exercism tracks.

Using the Test Runner Template

1. Ensure that your track has not already implemented a test runner. If there is, there will be a https://github.com/exercism/<track>-test-runner repository (i.e. if your track's slug is python, the test runner repo would be https://github.com/exercism/python-test-runner)
2. Follow GitHub's documentation for creating a repository from a template repository
   • Name your new repository based on your language track's slug (i.e. if your track is for Python, your test runner repo name is python-test-runner)
3. Remove this Exercism Test Runner Template section from the README.md file
4. Build the test runner, conforming to the Test Runner interface specification.
   • Update the files to match your track's needs. At the very least, you'll need to update bin/run.sh, Dockerfile and the test solutions in the tests directory
   • Tip: look for TODO: comments to point you towards code that need updating
   • Tip: look for OPTIONAL: comments to point you towards code that could be useful

Once you're happy with your test runner, open an issue on the exercism/exercism to request an official test runner repository for your track.

Exercism GDScript Test Runner

The Docker image to automatically run tests on GDScript solutions submitted to Exercism.

Test runner file format

There is currently no built-in testing framework available for Godot Engine. Because of that, Exercism uses a custom test runner for its GDScript track.

Each test suite consists of a single GDScript file. Its name has to be equal to the name of the tested file, with a _test suffix (so e.g. for example_success.gd the test file is called example_success_test.gd). The test runner will load this file and run all of the methods starting with test_, in the same order that they are defined in the GDScript file. Each test method represents a single test case, and the name of the method will be included in the results.json file as the name of the test.

Each test case will be called with a single argument. That argument is the solution script, prepared by the user and loaded by the test runner as a Script object. The test case can call any methods of the solution script, as well as perform any setup steps (if necessary).

Each test case is expected to return an Array of 2 elements. The first is the value received from calling a user defined method, the second is the expected value. The test runner will take care of comparing those values and generating an error message, if necessary.

A full test suite file might look like this:

func test_add_1_and_2(solution_script):
	return [solution_script.add_2_numbers(1, 2), 3]


func test_add_10_and_20(solution_script):
	return [solution_script.add_2_numbers(10, 20), 30]

Run the test runner

To run the tests of an arbitrary exercise, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run.sh <exercise-slug> <solution-dir> <output-dir>

Once the test runner has finished, its results will be written to <output-dir>/results.json.

Run the test runner on an exercise using Docker

This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment.

To run the tests of an arbitrary exercise using the Docker image, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-in-docker.sh <exercise-slug> <solution-dir> <output-dir>

Once the test runner has finished, its results will be written to <output-dir>/results.json.

Run the tests

To run the tests to verify the behavior of the test runner, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-tests.sh

These are golden tests that compare the results.json generated by running the current state of the code against the "known good" tests/<test-name>/results.json. All files created during the test run itself are discarded.

When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new tests/<test-name>/results.json file.

Run the tests using Docker

This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment.

To run the tests to verify the behavior of the test runner using the Docker image, do the following:

1. Open a terminal in the project's root
2. Run ./bin/run-tests-in-docker.sh

These are golden tests that compare the results.json generated by running the current state of the code against the "known good" tests/<test-name>/results.json. All files created during the test run itself are discarded.

When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new tests/<test-name>/results.json file.