[Exegesis] Add the ability to dry-run the measurement phase #121991
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This flag will make llvm-exegesis run everything except the actual snippet execution.
|
@llvm/pr-subscribers-llvm-binary-utilities Author: Min-Yih Hsu (mshockwave) ChangesThis flag will make llvm-exegesis run everything except the actual snippet execution. It is useful when we want to test some parts of the code between the Full diff: https://github.com/llvm/llvm-project/pull/121991.diff 3 Files Affected:
diff --git a/llvm/docs/CommandGuide/llvm-exegesis.rst b/llvm/docs/CommandGuide/llvm-exegesis.rst
index 8266d891a5e6b1..cd0bce9e2dbcc3 100644
--- a/llvm/docs/CommandGuide/llvm-exegesis.rst
+++ b/llvm/docs/CommandGuide/llvm-exegesis.rst
@@ -449,6 +449,11 @@ OPTIONS
crash when hardware performance counters are unavailable and for
debugging :program:`llvm-exegesis` itself.
+.. option:: --dry-run-measurement
+ If set, llvm-exegesis runs everything except the actual snippet execution.
+ This is useful if we want to test some part of the code without actually
+ running on native platforms.
+
.. option:: --execution-mode=[inprocess,subprocess]
This option specifies what execution mode to use. The `inprocess` execution
diff --git a/llvm/test/tools/llvm-exegesis/dry-run-measurement.test b/llvm/test/tools/llvm-exegesis/dry-run-measurement.test
new file mode 100644
index 00000000000000..82857e7998b5e6
--- /dev/null
+++ b/llvm/test/tools/llvm-exegesis/dry-run-measurement.test
@@ -0,0 +1,11 @@
+# RUN: llvm-exegesis --mtriple=riscv64 --mcpu=sifive-p470 --mode=latency --opcode-name=ADD --use-dummy-perf-counters --dry-run-measurement | FileCheck %s
+# REQUIRES: riscv-registered-target
+
+# This test makes sure that llvm-exegesis doesn't execute "cross-compiled" snippets in the presence of
+# --dry-run-measurement. RISC-V was chosen simply because most of the time we run tests on X86 machines.
+
+# Should not contain misleading results.
+# CHECK: measurements: []
+
+# Should not contain error messages like "snippet crashed while running: Segmentation fault".
+# CHECK: error: ''
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp b/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp
index a7771b99e97b1a..9b978c558c1fe7 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp
@@ -53,6 +53,12 @@
namespace llvm {
namespace exegesis {
+static cl::opt<bool>
+ DryRunMeasurement("dry-run-measurement",
+ cl::desc("Run every steps in the measurement phase "
+ "except executing the snippet."),
+ cl::init(false), cl::Hidden);
+
BenchmarkRunner::BenchmarkRunner(const LLVMState &State, Benchmark::ModeE Mode,
BenchmarkPhaseSelectorE BenchmarkPhaseSelector,
ExecutionModeE ExecutionMode,
@@ -140,13 +146,21 @@ class InProcessFunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
Scratch->clear();
{
auto PS = ET.withSavedState();
+ // We can't directly capture DryRunMeasurement in the lambda below.
+ bool DryRun = DryRunMeasurement;
CrashRecoveryContext CRC;
CrashRecoveryContext::Enable();
- const bool Crashed = !CRC.RunSafely([this, Counter, ScratchPtr]() {
- Counter->start();
- this->Function(ScratchPtr);
- Counter->stop();
- });
+ const bool Crashed =
+ !CRC.RunSafely([this, Counter, ScratchPtr, DryRun]() {
+ if (DryRun) {
+ Counter->start();
+ Counter->stop();
+ } else {
+ Counter->start();
+ this->Function(ScratchPtr);
+ Counter->stop();
+ }
+ });
CrashRecoveryContext::Disable();
PS.reset();
if (Crashed) {
|
|
@llvm/pr-subscribers-tools-llvm-exegesis Author: Min-Yih Hsu (mshockwave) ChangesThis flag will make llvm-exegesis run everything except the actual snippet execution. It is useful when we want to test some parts of the code between the Full diff: https://github.com/llvm/llvm-project/pull/121991.diff 3 Files Affected:
diff --git a/llvm/docs/CommandGuide/llvm-exegesis.rst b/llvm/docs/CommandGuide/llvm-exegesis.rst
index 8266d891a5e6b1..cd0bce9e2dbcc3 100644
--- a/llvm/docs/CommandGuide/llvm-exegesis.rst
+++ b/llvm/docs/CommandGuide/llvm-exegesis.rst
@@ -449,6 +449,11 @@ OPTIONS
crash when hardware performance counters are unavailable and for
debugging :program:`llvm-exegesis` itself.
+.. option:: --dry-run-measurement
+ If set, llvm-exegesis runs everything except the actual snippet execution.
+ This is useful if we want to test some part of the code without actually
+ running on native platforms.
+
.. option:: --execution-mode=[inprocess,subprocess]
This option specifies what execution mode to use. The `inprocess` execution
diff --git a/llvm/test/tools/llvm-exegesis/dry-run-measurement.test b/llvm/test/tools/llvm-exegesis/dry-run-measurement.test
new file mode 100644
index 00000000000000..82857e7998b5e6
--- /dev/null
+++ b/llvm/test/tools/llvm-exegesis/dry-run-measurement.test
@@ -0,0 +1,11 @@
+# RUN: llvm-exegesis --mtriple=riscv64 --mcpu=sifive-p470 --mode=latency --opcode-name=ADD --use-dummy-perf-counters --dry-run-measurement | FileCheck %s
+# REQUIRES: riscv-registered-target
+
+# This test makes sure that llvm-exegesis doesn't execute "cross-compiled" snippets in the presence of
+# --dry-run-measurement. RISC-V was chosen simply because most of the time we run tests on X86 machines.
+
+# Should not contain misleading results.
+# CHECK: measurements: []
+
+# Should not contain error messages like "snippet crashed while running: Segmentation fault".
+# CHECK: error: ''
diff --git a/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp b/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp
index a7771b99e97b1a..9b978c558c1fe7 100644
--- a/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp
+++ b/llvm/tools/llvm-exegesis/lib/BenchmarkRunner.cpp
@@ -53,6 +53,12 @@
namespace llvm {
namespace exegesis {
+static cl::opt<bool>
+ DryRunMeasurement("dry-run-measurement",
+ cl::desc("Run every steps in the measurement phase "
+ "except executing the snippet."),
+ cl::init(false), cl::Hidden);
+
BenchmarkRunner::BenchmarkRunner(const LLVMState &State, Benchmark::ModeE Mode,
BenchmarkPhaseSelectorE BenchmarkPhaseSelector,
ExecutionModeE ExecutionMode,
@@ -140,13 +146,21 @@ class InProcessFunctionExecutorImpl : public BenchmarkRunner::FunctionExecutor {
Scratch->clear();
{
auto PS = ET.withSavedState();
+ // We can't directly capture DryRunMeasurement in the lambda below.
+ bool DryRun = DryRunMeasurement;
CrashRecoveryContext CRC;
CrashRecoveryContext::Enable();
- const bool Crashed = !CRC.RunSafely([this, Counter, ScratchPtr]() {
- Counter->start();
- this->Function(ScratchPtr);
- Counter->stop();
- });
+ const bool Crashed =
+ !CRC.RunSafely([this, Counter, ScratchPtr, DryRun]() {
+ if (DryRun) {
+ Counter->start();
+ Counter->stop();
+ } else {
+ Counter->start();
+ this->Function(ScratchPtr);
+ Counter->stop();
+ }
+ });
CrashRecoveryContext::Disable();
PS.reset();
if (Crashed) {
|
mshockwave
changed the title
--dry-run-measurement
Done.
yeah I found it more difficult to support this in the subprocess mode, so I simply return an error message if the user try to dry-run in that mode. |
Could you give some examples of what you're talking about here? Debugging the benchmark runners? |
Actually #121993 is a good example (and why it's stacked on top of this PR): without |
boomanaiden154
approved these changes
Jan 9, 2025
mshockwave
added a commit
to mshockwave/llvm-project
that referenced
this pull request
Jan 9, 2025
…lvm#121991)" This reverts commit f8f8598.
mshockwave
added a commit
that referenced
this pull request
Jan 9, 2025
#122371) …#121991)" This reverts commit f8f8598. This breaks ARMv7 and s390x buildbot with the following message: ``` llvm-exegesis error: No available targets are compatible with triple "armv8l-unknown-linux-gnueabihf" FileCheck error: '<stdin>' is empty. FileCheck command line: /home/tcwg-buildbot/worker/clang-armv7-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-armv7-2stage/llvm/llvm/test/tools/llvm-exegesis/dry-run-measurement.test ```
BaiXilin
pushed a commit
to BaiXilin/llvm-fix-vnni-instr-types
that referenced
this pull request
Jan 12, 2025
llvm#122371) …llvm#121991)" This reverts commit f8f8598. This breaks ARMv7 and s390x buildbot with the following message: ``` llvm-exegesis error: No available targets are compatible with triple "armv8l-unknown-linux-gnueabihf" FileCheck error: '<stdin>' is empty. FileCheck command line: /home/tcwg-buildbot/worker/clang-armv7-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-armv7-2stage/llvm/llvm/test/tools/llvm-exegesis/dry-run-measurement.test ```
Mel-Chen
pushed a commit
to Mel-Chen/llvm-project
that referenced
this pull request
Jan 13, 2025
llvm#122371) …llvm#121991)" This reverts commit f8f8598. This breaks ARMv7 and s390x buildbot with the following message: ``` llvm-exegesis error: No available targets are compatible with triple "armv8l-unknown-linux-gnueabihf" FileCheck error: '<stdin>' is empty. FileCheck command line: /home/tcwg-buildbot/worker/clang-armv7-2stage/stage2/bin/FileCheck /home/tcwg-buildbot/worker/clang-armv7-2stage/llvm/llvm/test/tools/llvm-exegesis/dry-run-measurement.test ```
mshockwave
added a commit
that referenced
this pull request
Jan 13, 2025
…121991)" (#122775) This relands f8f8598 Follow up on #122371: The problem here is a little subtle: when we dry-run the measurement phase, we create a LLJIT instance without actually executing the snippets. The key is, LLJIT has its own TargetMachine which uses triple designated by LLVM_TARGET_ARCH (which is default to host). On a machine that does not support Exegesis, the LLJIT would fail to create its TargetMachine because llvm-exegesis don't even register the host's target! Putting this test into any of the target-specific folder won't help, because it's about the host. And personally I don't really want to use `exegesis-can-execute-<arch>` for generic tests like this -- it's too strict as we don't actually need to execute the snippet. My solution here is creating another test feature which is added only when LLVM_TARGET_ARCH is supported by llvm-exegesis. This feature is something in between `<arch>-registered-target` and `exegesis-can-execute-<arch>`.
mshockwave
added a commit
that referenced
this pull request
Jan 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With the new benchmark phase,
dry-run-measurement, llvm-exegesis can run everything except the actual snippet execution. It is useful when we want to test some parts of the code between theassemble-measured-codeandmeasurephase without actually running on native platforms.