diff --git a/CHANGELOG.md b/CHANGELOG.md index f42ae706..af6e9924 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,105 @@ ## [Unreleased] +## [1.3.0] Env Vars part of RunId, and Support for Machine-Specific Settings - 2025-02-03 + +This release has three major changes that will affect how ReBench interprets configurations, +and a change for `denoise` that requires an update of the `sudoers` file. + +### 1. "Run" identity is now based on all details, e.g. also env vars (#271) + +This change can affect the number of different runs ReBench identifies. +Until now, the identity of a "run" was based on the command line generated from the +configuration. However, this meant, we could not distinguish between runs +with for instance different environment variables. +From now on, the identity of runs is based on all details of the configuration, +which may mean that configuration files may result in more distinct runs +than they did previously. + +### 2. Build commands are combined and run as a single shell script (#269) + +The `build:` key takes a list of commands, which are now combined into a single +shell script. Previously, each command was run separately. +Since we avoid executing the same command multiple times, +treating commands separately led to confusing results and was not very useful. +By combining the commands into a single script, identifying redundant script +execution leads to more predictable results. + +### 3. Support for machine-specific settings (#272) + +A new lowest-priority configuration level is introduced, which is meant to define +machine-specific settings. This can be useful, for instance for settings like this: + +```yaml +machines: + smaller-machine: + cores: [1, 2] + larger-machine: + cores: [1, 2, 4, 8] +``` + +These settings are combined into a configuration based on the usual rules. +So, if a benchmark suite or a benchmark give specific values for `cores:`, these +have higher priority. However, one would likely use it, avoiding +benchmark-specific values. + +When executing ReBench on the corresponding machine, these settings can be +selected with the `-m` option. Thus, the larger machine configuration is used +when running: + +```bash +rebench -m larger-machine rebench.conf +``` + +### 4. `rebench` uses `denoise.py` directly as script (#281) + +Previously, we used the `rebench-denoise` script that is automatically created +during the setup process. However, this required a setup where ReBench was +installed so that the root user had access to it with a proper `PYTHONPATH`. +To avoid issues with setup, avoid having to pass `PYTHONPATH` through `sudo`, +and to make the use of denoise more robust, we changed `denoise.py` so that it +can be used directly as a script by ReBench. + +In practice, the means, it's likely that the `sudoers` file will need to be +updated to point to the `denoise.py` script directly. ReBench will output the +path to the script when it is not able to use it directly. + + +### Other Minor Changes + +#### Features + - add support for ReBenchDB API Version 2 (#236) + - expand `~` in paths, right before invocation (#240, #283) + - add summary for columns where all values are the same (#256) + - show `env` in debug output (#280) + +#### Changes + - make output for currently running benchmark more compact (#237) + - invoke denoise with absolute path to avoid need for having it on a secure_path (#238, #273) + - rename the machines variable/filters to a tags variable/filter (#264) + - support Python 3.13 as latest version, drop support for Python 3.8 (#268) + - reduce number of git invocations to get source information (#275) + - distinguish `rebench` return codes for different errors (#282) + +#### Bug Fixes + - added missing machine column heading for summary table (#246) + - behave more gracefully on bare-bone setup, e.g. without git (#245) + - raise error in `TimeAdapter` to match other adapters (#254) + - make profiling with `perf` more robust (#255) + - make denoise more robust to absent tools and running as root with a user-level installation (#260) + - fix handling of ctrl-c interrupts (#262) + - avoid unnecessary warning about data reporting, when no data is to be reported (#277) + +#### Development + - use black formatting (#267) + - use mypy type checking and add some annotations (#270) + - use `pip install —editable` in CI to avoid incorrect coverage reporting (#279) + - add testing on Rocky Linux with integration test (#282) + +Thanks to @antonzhukovin, @vext01, and @martinmcclure for their contributions! + +**Full Changelog**: https://github.com/smarr/ReBench/compare/v1.2.0...v1.3.0 + ## [1.2.0] Custom Gauge Adapters - 2023-08-06 The main feature of this release is the new support for custom @@ -20,7 +119,7 @@ Other new features: - make the current invocation accessible in the command as `%(invocation)s` (#230) Other changes: - - fix bug where `'None'` instead of `null` was reported to ReBenchDB (#232) + - make sure `null` is not reported as `'None'` to ReBenchDB (#232) - fix handling of environment variables when sudo is used (#210) - try `gtime` from MacPorts as alternative `time` command on macOS (#212) - update py-cpuinfo to work on macOS with ARM-base CPUs (#212) @@ -29,6 +128,8 @@ Other changes: Thanks to @naomiGrew for the contributions! +**Full Changelog**: https://github.com/smarr/ReBench/compare/v1.1.0...v1.2.0 + ## [1.1.0] Denoise - 2023-02-21 This release focuses on reducing the noise from the system (#143, #144). @@ -68,9 +169,9 @@ Other notable improvements: - make gauge adapter names in configurations case-insensitive (#202) - improve documentation (#197, #198) - use PyTest for unit tests (#192) - + **Full Changelog**: https://github.com/smarr/ReBench/compare/v1.0.1...v1.1.0 - + ## [1.0.1] - 2020-06-23 @@ -83,7 +184,7 @@ This is a bug fix release. This is the first official release of ReBench as a "feature-complete" product. Feature-complete here means, it is a tried and tested tool for benchmark -execution. It is highly +execution. It is highly [configurable](https://rebench.readthedocs.io/en/latest/config/), [documented](https://rebench.readthedocs.io/en/latest/), and [successfully used](https://github.com/smarr/ReBench#use-in-academia). @@ -136,7 +237,7 @@ Thank you! - added testing of Python 3.7 (#121) and ruamel.yaml (#123) - ensure config is YAML 1.2 compliant (#123) - added support for ReBenchDB (#129, #130) - + - fixed issues with error reporting (#128) - fixed handling of input size configuration (#117) @@ -145,7 +246,7 @@ Thank you! - added `--setup-only` option, to run one benchmark for each setup (#110, #115) - added `ignore_timeout` setting to accept known timeouts without error (#118) - added `retries_after_failure` setting (#107, #108) - + - fixed data loading, which ignored warmup setting (#111, #116) - fixed how settings are inherited for follow documentation (#112, #113) - fixed message for consecutive failures (#109) @@ -219,7 +320,8 @@ Thank you! - [0.6.0] - 2014-05-19 - [0.5.0] - 2014-03-25 -[Unreleased]: https://github.com/smarr/ReBench/compare/v1.2.0...HEAD +[Unreleased]: https://github.com/smarr/ReBench/compare/v1.3.0...HEAD +[1.3.0]: https://github.com/smarr/ReBench/compare/v1.2.0...v1.3.0 [1.2.0]: https://github.com/smarr/ReBench/compare/v1.1.0...v1.2.0 [1.1.0]: https://github.com/smarr/ReBench/compare/v1.0.1...v1.1.0 [1.0.1]: https://github.com/smarr/ReBench/compare/v1.0.0...v1.0.1 diff --git a/docs/config.md b/docs/config.md index 83d1132f..d67a7568 100644 --- a/docs/config.md +++ b/docs/config.md @@ -580,14 +580,14 @@ A list of commands/strings to be executed by the system's shell. They are intended to set up the system for benchmarking, typically to build binaries, compiled archives, etc. -Each command is executed once before any benchmark that depends on it -is executed. If the `location` of the suite is set, it is used as +The list of command is combined into a single shell script, which +is executed once before any benchmark that depends on it +runs. If the `location` of the suite is set, it is used as working directory. Otherwise, it is the current working directory of ReBench. -`build:` is a list of commands to allow multiple suites and executors to depend on the -same build command without executing it multiple times. -For this purpose, build commands are considered the same when they have the -same command and location (based on simple string comparisons). +When multiple suites and executors depend on lists of commands, +where the combined script is identical, and to be executed in the same location, +the script is executed only once. Commands are executed with an environment managed by ReBench. The environment starts empty, but can be added to using the [`env`](#env) directive. diff --git a/docs/release.md b/docs/release.md index 0a625088..4516e0c2 100644 --- a/docs/release.md +++ b/docs/release.md @@ -3,7 +3,7 @@ This list is supposed to make sure we do not forget any important steps during release. -1. Check and update issue tracker for issues associated with current milestone +1. Check and update issue tracker for issues that should be included in the release. 2. Make sure all desired changes are merged with `master`. diff --git a/rebench/__init__.py b/rebench/__init__.py index 9fefac79..67bc602a 100644 --- a/rebench/__init__.py +++ b/rebench/__init__.py @@ -1 +1 @@ -__version__ = "1.3.0.dev2" +__version__ = "1.3.0"