Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.verifier.expect file cause compilation tests to be ignored #4355

Closed
RustanLeino opened this issue Jul 29, 2023 · 0 comments · Fixed by #4356
Closed

.verifier.expect file cause compilation tests to be ignored #4355

RustanLeino opened this issue Jul 29, 2023 · 0 comments · Fixed by #4356
Assignees
Labels
area: build-system Support for dependencies in Dafny, generation of target language build files kind: bug Crashes, unsoundness, incorrect output, etc. If possible, add a `part:` label misc: tests New tests or tutorials part: tools Tools built on top of Dafny

Comments

@RustanLeino
Copy link
Collaborator

Dafny version

4.2.0

Code to produce this issue

===== TestIt.dfy
// RUN: %testDafnyForEachCompiler "%s"

method Main() {
  ghost var n := 15;
  if j :| 0 <= j < n { // this gives a no-trigger warning
  }
  print "hello\n";
}
===== TestIt.dfy.verifier.expect
TestIt.dfy(5,5): Warning: /!\ No terms found to trigger on.
===== TestIt.dfy.expect
good bye

Command to run and resulting output

DAFNY/Test/dafny0> lit TestIt.dfy
...
-- Testing: 1 tests, 1 workers --
PASS: Dafny :: dafny0/TestIt.dfy (1 of 1)

Testing Time: 2.07s
  Passed: 1

What happened?

When a test in the Dafny test suite has a .verifier.expect file that consists of only warning messages, then TestDafny for-each-compiler never calls the compiler.

As demonstrated by the repro above, the compiled program prints hello and the .expect file expects good bye. Yet, the test passes.

What type of operating system are you experiencing the problem on?

Mac

@RustanLeino RustanLeino added kind: bug Crashes, unsoundness, incorrect output, etc. If possible, add a `part:` label misc: tests New tests or tutorials part: tools Tools built on top of Dafny area: build-system Support for dependencies in Dafny, generation of target language build files labels Jul 29, 2023
@RustanLeino RustanLeino self-assigned this Jul 29, 2023
RustanLeino added a commit that referenced this issue Aug 6, 2023
…4356)

This PR fixes and changes the behavior of `%testDafnyForEachCompiler`
tests. The bug that is fixed had caused 16 compiler tests in the Dafny
test suite to not run.

Here is a description of the new behavior, where _italicized_ parts
indicate changes from before:

* Verification is expected to go through with a 0 exit code.
* If there is a `.verifier.expect` file, then the actual output is
expected to contain a blank line followed by a line that starts with
`Dafny program verifier`, and this portion of the output will be removed
before comparing the output. The remaining output is compared with the
contents of the `.verifier.expect` file and is expected to match.
* If there is no `.verifier.expect` file, the verifier output minus the
`Dafny program verifier` line (see previous bullet) is expected to be
empty.
* When running the test for each compiler, if the output contains a line
starting with `Dafny program verifier`, then _everything up to and
include that line is removed_ before comparing the output.

This PR adds add test in `Test/metatests`.

Fixes #4355

<small>By submitting this pull request, I confirm that my contribution
is made under the terms of the [MIT
license](https://github.com/dafny-lang/dafny/blob/master/LICENSE.txt).</small>

---------

Co-authored-by: Robin Salkeld <[email protected]>
keyboardDrummer pushed a commit to keyboardDrummer/dafny that referenced this issue Sep 15, 2023
…afny-lang#4356)

This PR fixes and changes the behavior of `%testDafnyForEachCompiler`
tests. The bug that is fixed had caused 16 compiler tests in the Dafny
test suite to not run.

Here is a description of the new behavior, where _italicized_ parts
indicate changes from before:

* Verification is expected to go through with a 0 exit code.
* If there is a `.verifier.expect` file, then the actual output is
expected to contain a blank line followed by a line that starts with
`Dafny program verifier`, and this portion of the output will be removed
before comparing the output. The remaining output is compared with the
contents of the `.verifier.expect` file and is expected to match.
* If there is no `.verifier.expect` file, the verifier output minus the
`Dafny program verifier` line (see previous bullet) is expected to be
empty.
* When running the test for each compiler, if the output contains a line
starting with `Dafny program verifier`, then _everything up to and
include that line is removed_ before comparing the output.

This PR adds add test in `Test/metatests`.

Fixes dafny-lang#4355

<small>By submitting this pull request, I confirm that my contribution
is made under the terms of the [MIT
license](https://github.com/dafny-lang/dafny/blob/master/LICENSE.txt).</small>

---------

Co-authored-by: Robin Salkeld <[email protected]>
keyboardDrummer pushed a commit that referenced this issue Sep 19, 2023
…4356)

This PR fixes and changes the behavior of `%testDafnyForEachCompiler`
tests. The bug that is fixed had caused 16 compiler tests in the Dafny
test suite to not run.

Here is a description of the new behavior, where _italicized_ parts
indicate changes from before:

* Verification is expected to go through with a 0 exit code.
* If there is a `.verifier.expect` file, then the actual output is
expected to contain a blank line followed by a line that starts with
`Dafny program verifier`, and this portion of the output will be removed
before comparing the output. The remaining output is compared with the
contents of the `.verifier.expect` file and is expected to match.
* If there is no `.verifier.expect` file, the verifier output minus the
`Dafny program verifier` line (see previous bullet) is expected to be
empty.
* When running the test for each compiler, if the output contains a line
starting with `Dafny program verifier`, then _everything up to and
include that line is removed_ before comparing the output.

This PR adds add test in `Test/metatests`.

Fixes #4355

<small>By submitting this pull request, I confirm that my contribution
is made under the terms of the [MIT
license](https://github.com/dafny-lang/dafny/blob/master/LICENSE.txt).</small>

---------

Co-authored-by: Robin Salkeld <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: build-system Support for dependencies in Dafny, generation of target language build files kind: bug Crashes, unsoundness, incorrect output, etc. If possible, add a `part:` label misc: tests New tests or tutorials part: tools Tools built on top of Dafny
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant