OpenSBT - A Modular Framework for Search-based Testing of Automated Driving Systems

Table of Contents

• About
• Overview
• Installation
• Getting Started
• Output
• Application
• Contribution
• Acknowledgements
• Reference
• License

About

OpenSBT is a modular and extensible codebase designed to facilitate search-based testing of automated driving systems, whether for research, in education or in an industrial context. It provides well-defined interfaces for the integration of search algorithms, fitness functions, testing oracles, and simulation environments in a modular and flexible manner. In addition, OpenSBT supports the visualization of testing results and analysis of the failing behaviour of automated driving systems.

Overview

OpenSBT builds upon Pymoo 0.6.0.1 and extends internal optimization related models to tailor heuristic search algorithms for testing automated driving systems. An introductory video of OpenSBT can be found here: https://www.youtube.com/watch?v=qi_CTTzrk5s.

Installation

OpenSBT requires Python to be installed and its compatibility has been tested with Python 3.8. OpenSBT can be run as a standalone application or can be imported as a library.

You can install OpenSBT via pypi:

pip install opensbt

If using as a standalone appplication install the dependencies after downloading OpenSBT files:

pip install -r requirements.txt

A complete installation example is available as a jupyter notebook.

Getting Started

In the core implementation of OpenSBT a simplified dummy simulator is integrated (linear motion planning, no GPU required). To run an example test generation for this example run (flag -e holds the experiment number):

python run.py -e 5

Several result artefacts are generated after the generation has finished. All artefacts are written into the results folder in a folder named DummySimulator (problem name).

You can find several tutorials as jupyter notebooks which explain step-by-step of how integrate custom simulators and define testing components and objectives in OpenSBT.

The tutorials include as an example the integration of a real-world FMI-based AEB agent developed in the fortiss Mobility lab simulated in CARLA using the simulator adapter CARLA Runner Extension.

As another example we have integrated Simulink-based systems simulated in Prescan simulator adapter into OpenSBT.

Output

OpenSBT produces several result artefacts. An excerpt is given below.

Type	Description	Example
Design Space Plot	Visualization of all evaluated test cases in the input space and predicted failing regions using the decision tree algorithm, pairwise. Constraints of derived regions are stored in a CSV file bounds_regions.csv and the learned tree in tree.pdf
Scenario 2D Visualization	Visualization of traces of the ego vehicle and adversaries in a two-dimensional GIF animation
Objective Space Plot	Visualization of fitness values of evaluated test cases, pairwise.
All Testcases	CSV file of all test inputs of all evaluated testcases	all_testcases.csv
All Failing Testcases	CSV file of all failing test inputs of all evaluated testcases	all_critical_testcases.csv
Calculation Properties	CSV file of all experiment configuration parameters (e.g. algorithm parameters, such as population size, number iterations; search space, fitness function etc..).	calculation_properties.csv
Evaluation Results	CSV file containing performance values of the algorithm, e.g., number failing test cases found in relation to all evaluations, execution time.	summary_results.csv

Application

Following Simulators have been integrated already into OpenSBT:

• Prescan: Example
• CARLA: Example
• Donkey: Example
• Udacity: Example
• BeamNG: Example
• esmini: Example

OpenSBT has been applied in research:

• Replication Study: Paper, Code
• Autonomous Driving Testing: Paper, Video
• Algorithm Benchmarking: Paper, Code
• Failure Coverage Study: Paper, Code

Contribution

If you like to contribute please contact one of the developers listed below or create a pull request. If you face any issues with OpenSBT feel free to create an issue or contact the developers.

Acknowledgements

OpenSBT has been developed by Lev Sorokin, Tiziano Munaro and Damir Safin within the FOCETA Project. Special thanks go to Brian Hsuan-Cheng Liao and Adam Molin from DENSO AUTOMOTIVE Deutschland GmbH for their valuable feedback and evaluation of OpenSBT on the AVP Case Study in the Prescan simulator.

Reference

If you use or extend OpenSBT please cite our framework. Here is an example BibTeX entry:

@inproceedings{10.1145/3639478.3640027,
    author = {Sorokin, Lev and Munaro, Tiziano and Safin, Damir and Liao, Brian Hsuan-Cheng and Molin, Adam},
    title = {OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems},
    year = {2024},
    isbn = {9798400705021},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3639478.3640027},
    doi = {10.1145/3639478.3640027},
    booktitle = {Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings},
    pages = {94–98},
    numpages = {5},
    keywords = {search-based software testing, metaheuristics, scenario-based testing, autonomous driving, automated driving},
    location = {, Lisbon, Portugal, },
    series = {ICSE-Companion '24}
}

License

OpenSBT is licensed under the Apache License, Version 2.0.