Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run inline tests in parallel #7012

Merged
merged 2 commits into from
Mar 20, 2023
Merged

Run inline tests in parallel #7012

merged 2 commits into from
Mar 20, 2023

Conversation

hhugo
Copy link
Collaborator

@hhugo hhugo commented Feb 6, 2023

Fix #1516.
FIX #6942
Based on #6933 (merged)

  • inline_tests backends can now specify flags used to list all partitions.
  • for each mode, we generate .$(libname).inline-tests/partitions-$(mode) that contains the list of partition of the lib and mode.
  • Each partition runs concurrently (using concurrent)
  • After each partition has run, we start diffing against expected file concurrently.

https://github.com/hhugo/ppx_inline_test/commits/dune-3.8 shows how ppx_inline_tests can be fixed to support the new feature.

@hhugo hhugo requested a review from christinerose as a code owner February 6, 2023 16:20
@hhugo hhugo marked this pull request as draft February 6, 2023 16:20
@hhugo hhugo changed the title Inline tests para Run inline tests in parallel Feb 6, 2023
Copy link
Collaborator

@christinerose christinerose left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very minor grammar/syntax/formatting suggestions. :)

doc/concepts.rst Outdated Show resolved Hide resolved
test/blackbox-tests/test-cases/actions/conc.t Outdated Show resolved Hide resolved
test/blackbox-tests/test-cases/actions/conc.t Outdated Show resolved Hide resolved
@Alizter
Copy link
Collaborator

Alizter commented Feb 6, 2023

@christinerose Those are from my PR <3

@hhugo
Copy link
Collaborator Author

hhugo commented Feb 6, 2023

@Alizter I'm not sure about concurrent diff. Does dune prevent interleaving of output ?

@rgrinberg
Copy link
Member

Does dune prevent interleaving of output ?

Yes

@hhugo hhugo force-pushed the inline-tests-para branch from 20f2ce8 to 5c5a525 Compare February 6, 2023 18:32
@rgrinberg rgrinberg linked an issue Feb 6, 2023 that may be closed by this pull request
in
action
in
let+ action = actions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to use conc on these actions. You can just add them individually to the alias for the same effect. It's only within a single partition that you need to use conc to call diff on all the sources inside that partition.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is currently not relationship between partitions and source files. We need to run all partition before we can diff sources

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Well you can still run them in parallel without the use of conc. I would suggest to do that as a first step and leave the conc stuff for a subsequent PR.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what your asking for here. Could you be more specific ? What actions are you talking about exactly ? Diffs need to run after the all the tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong, but this is what you're doing currently:

  1. Execute the runner to the get the list of partitions
  2. Generate a list of actions from this list of partitions
  3. Run these actions concurrently.
  4. Then diff them all concurrently

Now here's what I suggested:

  1. Execute the runner to get the list of partitions
  2. For every partition, create an action that will execute the partition. You can attach all these actions to some alias partitions in the inline tests directory
  3. Create another action that will depend on the partitions alias and will then run the diff (concurrently, once Ali's PR is merged). Add this action to the runtest alias.

The advantage of my proposal is that passing partitions will not be re-executed.

But now that I think about it, you might need to go through contortions to make sure that executing the test to get the list of partitions will be done when loading the .inline-tests sub directory rather than the directory where you're building the inline tests themselves. I'm not so sure it's worth it anymore.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The advantage of my proposal is that passing partitions will not be re-executed.

I'm not sure what you mean here.
Partitions will have have to be recomputed every time the executable is regenerated.
Maybe you mean that actions responsible for running each partition would not have to be recomputed ?

I think what you're proposing is exactly what I commented about in #1516 (comment)

I've tried to implement the feature and got stuck realizing that I want to declare an arbitrary number of alias action (add_alias_action) based on a list of partition previously computed. I think this cannot currently be done, is it correct ?

And I thought it was impossible, but maybe it's possible with contortions

@hhugo hhugo force-pushed the inline-tests-para branch from 0e1adeb to 3350fce Compare February 9, 2023 16:23
@hhugo hhugo marked this pull request as ready for review February 9, 2023 16:31
@hhugo
Copy link
Collaborator Author

hhugo commented Feb 9, 2023

I've rebased on #6933.

@hhugo hhugo added this to the 3.8.0 milestone Feb 10, 2023
@hhugo hhugo force-pushed the inline-tests-para branch 5 times, most recently from 584c19f to 692f641 Compare February 27, 2023 22:47
@hhugo
Copy link
Collaborator Author

hhugo commented Feb 27, 2023

#6933 is merged. Please review.

in
List.map ~f:(fun x -> Some x) partitions
in
List.map partitions_flags ~f:(fun p -> action mode (flags p))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intended to ask for all these deps to just compute the partitions file?

Also, should we just sandbox the computation of partitions by default?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intended to ask for all these deps to just compute the partitions file?

What do you mean by all these deps ?

Also, should we just sandbox the computation of partitions by default?

I don't know much about sandboxing in dune, How does one change the default ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by all these deps ?

I mean info.deps - the deps that are in (inline_tests (deps ..))

I don't know much about sandboxing in dune, How does one change the default ?

I can do this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could imagine that the content of some info.deps is used to compute the list of partitions.

let flags =
let partitions_flags : string list Action_builder.t option =
match
List.filter_map backends ~f:(fun backend ->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me again what are "backends" in this situations and why we can just append all their partition flags like this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backends are inline test backends, We append all their partition flags similar to what we already do with their other flags (see backend.Backend.info.flags).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Do we have any tests with more than one backend? It might be worth adding such a test for partitions as well.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just discovered we have test/blackbox-tests/test-cases/inline_tests/too-many-backends.t.
I'm guessing that the list can have multiple entries when backend extend each other. (like ppx_inline_tests and ppx_expect).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've extended the test to have 2 backends, one extending the other, and only the second one using the partitioning mechanism.

@hhugo
Copy link
Collaborator Author

hhugo commented Mar 1, 2023

I'm now trying to modify ppx_inline_tests/ppx_expect to support the partitioning in dune.
One need to specify the partition when preprocessing each ml file.

  • either with a flag -inline-test-lib
  • or a ppxlib cookie library-name

The argument should be of form ${library-name}:${partition-name} where partition name is usually the basename of the module.

@hhugo
Copy link
Collaborator Author

hhugo commented Mar 1, 2023

I've manage to have ppx_inline_tests (and ppx_expect) working with partitioning enabled with this commit hhugo/ppx_inline_test@c8e5d74.

I'm not sure about the new ppxlib cookie used to communicated partition names.

src/dune_rules/preprocessing.ml Outdated Show resolved Hide resolved
@hhugo
Copy link
Collaborator Author

hhugo commented Mar 1, 2023

I believe that the current state is enough to port ppx_inline_tests. See https://github.com/hhugo/ppx_inline_test/commits/dune-3.8

doc/tests.rst Outdated Show resolved Hide resolved
@hhugo hhugo force-pushed the inline-tests-para branch from dc358a2 to b9a8d06 Compare March 7, 2023 22:03
@hhugo
Copy link
Collaborator Author

hhugo commented Mar 7, 2023

I've just rebased this PR and extended the tests

@hhugo hhugo requested a review from rgrinberg March 7, 2023 22:06
@hhugo hhugo force-pushed the inline-tests-para branch from b9a8d06 to 33bb670 Compare March 7, 2023 22:19
Allow specifying partitions of inline tests. Partitions in the same test
suite will be executed concurrently.

Signed-off-by: Hugo Heuzard <[email protected]>
Signed-off-by: Rudi Grinberg <[email protected]>
@hhugo hhugo force-pushed the inline-tests-para branch from 33bb670 to fdea3fe Compare March 11, 2023 09:05
@hhugo
Copy link
Collaborator Author

hhugo commented Mar 14, 2023

Does this need more reviews or can we merge? @rgrinberg ?

@rgrinberg
Copy link
Member

Let me take a look

Copy link
Collaborator

@christinerose christinerose left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one typo, two verb tense errors, and some things that perhaps need to be in monospace (?)

doc/tests.rst Outdated Show resolved Hide resolved
@hhugo
Copy link
Collaborator Author

hhugo commented Mar 17, 2023

Let me take a look

@rgrinberg, Gentle ping

Co-authored-by: Christine Rose <[email protected]>
Signed-off-by: hhugo <[email protected]>
@rgrinberg rgrinberg merged commit 247664d into ocaml:main Mar 20, 2023
emillon added a commit to emillon/opam-repository that referenced this pull request Apr 18, 2023
CHANGES:

- When a rule's action is interrupted, delete any leftover directory targets.
  This is consistent with how we treat file targets. (@rgrinberg, 7564)

- Fix plugin loading with findlib. The functionality was broken in 3.7.0.
  (ocaml/dune#7556, @anmonteiro)

- Introduce a `public_headers` field on libraries. This field is like
  `install_c_headers`, but it allows to choose the extension and choose the
  paths for the installed headers. (ocaml/dune#7512, @rgrinberg)

- Load the host context `findlib.conf` when cross-compiling (ocaml/dune#7428, fixes
  ocaml/dune#1701, @rgrinberg, @anmonteiro)

- Resolve `ppx_runtime_libraries` in the target context when cross compiling
  (ocaml/dune#7450, fixes ocaml/dune#2794, @anmonteiro)

- Use `$PKG_CONFIG`, when set, to find the `pkg-config` binary  (ocaml/dune#7469, fixes
  ocaml/dune#2572, @anmonteiro)

- Preliminary support for Coq compiled intefaces (`.vos` files) enabled via
  `(mode vos)` in `coq.theory` stanzas. This can be used in combination with
  `dune coq top` to obtain fast re-building of dependencies (with no checking
  of proofs) prior to stepping into a file. (ocaml/dune#7406, @rlepigre)

- Fix dune crashing on MacOS in watch mode whenever `$PATH` contains `$PWD`
  (ocaml/dune#7441, fixes ocaml/dune#6907, @rgrinberg)

- Fix `dune install` when cross compiling (ocaml/dune#7410, fixes ocaml/dune#6191, @anmonteiro,
  @rizo)

- Find `pps` dependencies in the host context when cross-compiling,  (ocaml/dune#7410,
  fixes ocaml/dune#4156, @anmonteiro)

- Dune in watch mode no longer builds concurrent rules in serial (ocaml/dune#7395
  @rgrinberg, @jchavarri)

- `dune coq top` now correctly respects the project root when called from a
  subdirectory. However, absolute filenames passed to `dune coq top` are no
  longer supported (due to being buggy) (ocaml/dune#7357, fixes ocaml/dune#7344, @rlepigre and
  @Alizter)

- Added a `--no-build` option to `dune coq top` for avoiding rebuilds (ocaml/dune#7380,
  fixes ocaml/dune#7355, @Alizter)

- RPC: Ignore SIGPIPE when clients suddenly disconnect (ocaml/dune#7299, ocaml/dune#7319, fixes
  ocaml/dune#6879, @rgrinberg)

- Always clean up the UI on exit. (ocaml/dune#7271, fixes ocaml/dune#7142 @rgrinberg)

- Bootstrap: remove reliance on shell. Previously, we'd use the shell to get
  the number of processors. (ocaml/dune#7274, @rgrinberg)

- Bootstrap: correctly detect the number of processors by allowing `nproc` to be
  looked up in `$PATH` (ocaml/dune#7272, @Alizter)

- Speed up file copying on macos by using `clonefile` when available
  (@rgrinberg, ocaml/dune#7210)

- Adds support for loading plugins in toplevels (ocaml/dune#6082, fixes ocaml/dune#6081,
  @ivg, @richardlford)

- Support commands that output 8-bit and 24-bit colors in the terminal (ocaml/dune#7188,
  @Alizter)

- Speed up rule generation for libraries and executables with many modules
  (ocaml/dune#7187, @jchavarri)

- Add `--watch-exclusions` to Dune build options (ocaml/dune#7216, @jonahbeckford)

- Do not re-render UI on every frame if the UI doesn't change (ocaml/dune#7186, fix
  ocaml/dune#7184, @rgrinberg)

- Make coq_db creation in scope lazy (@ejgallego, ocaml/dune#7133)

- Non-user proccesses such as version control or config checking are now run
  silently. (ocaml/dune#6994, fixes ocaml/dune#4066, @Alizter)

- Add the `--display-separate-messages` flag to separate the error messages
  produced by commands with a blank line. (ocaml/dune#6823, fixes ocaml/dune#6158, @esope)

- Accept the Ordered Set Language for the `modes` field in `library` stanzas
  (ocaml/dune#6611, @anmonteiro).

- dune install now respects --display quiet mode (ocaml/dune#7116, fixes ocaml/dune#4573, fixes
  ocaml/dune#7106, @Alizter)

- Stub shared libraries (dllXXX_stubs.so) in Dune-installed libraries could not
  be used as dependencies of libraries in the workspace (eg when compiling to
  bytecode and/or Javascript).  This is now fixed. (ocaml/dune#7151, @nojb)

- Allow the main module of a library with `(stdlib ...)` to depend on other
  libraries (ocaml/dune#7154, @anmonteiro).

- Bytecode executables built for JSOO are linked with `-noautolink` and no
  longer depend on the shared stubs of their dependent libraries (ocaml/dune#7156, @nojb)

- Added a new user action `(concurrent )` which is like `(progn )` but runs the
  actions concurrently. (ocaml/dune#6933, @Alizter)

- Allow `(stdlib ...)` to be used with `(wrapped false)` in library stanzas
  (ocaml/dune#7139, @anmonteiro).

- Allow parallel execution of inline tests partitions (ocaml/dune#7012, @hhugo)

- Support `(link_flags ...)` in `(cinaps ...)` stanza. (ocaml/dune#7423, fixes ocaml/dune#7416,
  @nojb)

- Allow `(package ...)` in any position within `(rule ...)` stanza (ocaml/dune#7445,
  @Leonidas-from-XIV)

- Always include `opam` files in the generated `.install` file. Previously, it
  would not be included whenever `(generate_opam_files true)` was set and the
  `.install` file wasn't yet generated. (ocaml/dune#7547, @rgrinberg)
emillon added a commit to emillon/opam-repository that referenced this pull request May 23, 2023
CHANGES:

- Fix string quoting in the json file written by `--trace-file` (ocaml/dune#7773,
  @rleshchinskiy)

- Read `pkg-config` arguments from the `PKG_CONFIG_ARGN` environment variable
  (ocaml/dune#1492, ocaml/dune#7734, @anmonteiro)

- Correctly set `MANPATH` in `dune exec`. Previously, we would use the `bin/`
  directory of the context. (ocaml/dune#7655, @rgrinberg)

- Allow overriding the `ocaml` binary with findlib configuration (ocaml/dune#7648,
  @rgrinberg)

- merlin: ignore instrumentation settings for preprocessing. (ocaml/dune#7606, fixes
  ocaml/dune#7465, @Alizter)

- When a rule's action is interrupted, delete any leftover directory targets.
  This is consistent with how we treat file targets. (ocaml/dune#7564, @rgrinberg)

- Fix plugin loading with findlib. The functionality was broken in 3.7.0.
  (ocaml/dune#7556, @anmonteiro)

- Introduce a `public_headers` field on libraries. This field is like
  `install_c_headers`, but it allows to choose the extension and choose the
  paths for the installed headers. (ocaml/dune#7512, @rgrinberg)

- Load the host context `findlib.conf` when cross-compiling (ocaml/dune#7428, fixes
  ocaml/dune#1701, @rgrinberg, @anmonteiro)

- Add a `coqdoc_flags` field to the `coq.theory` stanza allowing the user to
  pass extra arguments to `coqdoc`. (ocaml/dune#7676, fixes ocaml/dune#7954 @Alizter)

- Resolve `ppx_runtime_libraries` in the target context when cross compiling
  (ocaml/dune#7450, fixes ocaml/dune#2794, @anmonteiro)

- Use `$PKG_CONFIG`, when set, to find the `pkg-config` binary  (ocaml/dune#7469, fixes
  ocaml/dune#2572, @anmonteiro)

- Modules that were declared in `(modules_without_implementation)`,
  `(private_modules)` or `(virtual_modules)` but not declared in `(modules)`
  will cause Dune to emit a warning which will become an error in 3.9. (ocaml/dune#7608,
  fixes ocaml/dune#7026, @Alizter)

- Preliminary support for Coq compiled intefaces (`.vos` files) enabled via
  `(mode vos)` in `coq.theory` stanzas. This can be used in combination with
  `dune coq top` to obtain fast re-building of dependencies (with no checking
  of proofs) prior to stepping into a file. (ocaml/dune#7406, @rlepigre)

- Fix dune crashing on MacOS in watch mode whenever `$PATH` contains `$PWD`
  (ocaml/dune#7441, fixes ocaml/dune#6907, @rgrinberg)

- Fix `dune install` when cross compiling (ocaml/dune#7410, fixes ocaml/dune#6191, @anmonteiro,
  @rizo)

- Find `pps` dependencies in the host context when cross-compiling,  (ocaml/dune#7410,
  fixes ocaml/dune#4156, @anmonteiro)

- Dune in watch mode no longer builds concurrent rules in serial (ocaml/dune#7395
  @rgrinberg, @jchavarri)

- Dune can now detect Coq theories from outside the workspace. This allows for
  composition with installed theories (not necessarily installed with Dune).
  (ocaml/dune#7047, @Alizter, @ejgallego)

- `dune coq top` now correctly respects the project root when called from a
  subdirectory. However, absolute filenames passed to `dune coq top` are no
  longer supported (due to being buggy) (ocaml/dune#7357, fixes ocaml/dune#7344, @rlepigre and
  @Alizter)

- Added a `--no-build` option to `dune coq top` for avoiding rebuilds (ocaml/dune#7380,
  fixes ocaml/dune#7355, @Alizter)

- RPC: Ignore SIGPIPE when clients suddenly disconnect (ocaml/dune#7299, ocaml/dune#7319, fixes
  ocaml/dune#6879, @rgrinberg)

- Always clean up the UI on exit. (ocaml/dune#7271, fixes ocaml/dune#7142 @rgrinberg)

- Bootstrap: remove reliance on shell. Previously, we'd use the shell to get
  the number of processors. (ocaml/dune#7274, @rgrinberg)

- Bootstrap: correctly detect the number of processors by allowing `nproc` to be
  looked up in `$PATH` (ocaml/dune#7272, @Alizter)

- Speed up file copying on macos by using `clonefile` when available
  (@rgrinberg, ocaml/dune#7210)

- Adds support for loading plugins in toplevels (ocaml/dune#6082, fixes ocaml/dune#6081,
  @ivg, @richardlford)

- Support commands that output 8-bit and 24-bit colors in the terminal (ocaml/dune#7188,
  @Alizter)

- Speed up rule generation for libraries and executables with many modules
  (ocaml/dune#7187, @jchavarri)

- Add `--watch-exclusions` to Dune build options (ocaml/dune#7216, @jonahbeckford)

- Do not re-render UI on every frame if the UI doesn't change (ocaml/dune#7186, fix
  ocaml/dune#7184, @rgrinberg)

- Make coq_db creation in scope lazy (@ejgallego, ocaml/dune#7133)

- Non-user proccesses such as version control or config checking are now run
  silently. (ocaml/dune#6994, fixes ocaml/dune#4066, @Alizter)

- Add the `--display-separate-messages` flag to separate the error messages
  produced by commands with a blank line. (ocaml/dune#6823, fixes ocaml/dune#6158, @esope)

- Accept the Ordered Set Language for the `modes` field in `library` stanzas
  (ocaml/dune#6611, @anmonteiro).

- dune install now respects --display quiet mode (ocaml/dune#7116, fixes ocaml/dune#4573, fixes
  ocaml/dune#7106, @Alizter)

- Stub shared libraries (dllXXX_stubs.so) in Dune-installed libraries could not
  be used as dependencies of libraries in the workspace (eg when compiling to
  bytecode and/or Javascript).  This is now fixed. (ocaml/dune#7151, @nojb)

- Allow the main module of a library with `(stdlib ...)` to depend on other
  libraries (ocaml/dune#7154, @anmonteiro).

- Bytecode executables built for JSOO are linked with `-noautolink` and no
  longer depend on the shared stubs of their dependent libraries (ocaml/dune#7156, @nojb)

- Added a new user action `(concurrent )` which is like `(progn )` but runs the
  actions concurrently. (ocaml/dune#6933, @Alizter)

- Allow `(stdlib ...)` to be used with `(wrapped false)` in library stanzas
  (ocaml/dune#7139, @anmonteiro).

- Allow parallel execution of inline tests partitions (ocaml/dune#7012, @hhugo)

- Support `(link_flags ...)` in `(cinaps ...)` stanza. (ocaml/dune#7423, fixes ocaml/dune#7416,
  @nojb)

- Allow `(package ...)` in any position within `(rule ...)` stanza (ocaml/dune#7445,
  @Leonidas-from-XIV)

- Always include `opam` files in the generated `.install` file. Previously, it
  would not be included whenever `(generate_opam_files true)` was set and the
  `.install` file wasn't yet generated. (ocaml/dune#7547, @rgrinberg)

- Fix regression where Merlin was unable to handle filenames with uppercase
  letters under Windows. (ocaml/dune#7577, @nojb)

- On nix+macos, pass `-f` to the codesign hook to avoid errors when the binary
  is already signed (ocaml/dune#7183, fixes ocaml/dune#6265, @greedy)

- Fix bug where RPC clients built with dune-rpc-lwt would crash when closing
  their connection to the server (ocaml/dune#7581, @gridbugs)

- Introduce mdx stanza 0.4 requiring mdx >= 2.3.0 which updates the default
  list of files to include `*.mld` files (ocaml/dune#7582, @Leonidas-from-XIV)

- Fix RPC server on Windows (used for OCaml-LSP). (ocaml/dune#7666, @nojb)

- Coq language versions less 0.8 are deprecated, and will be removed
  in an upcoming Dune version. All users are required to migrate to
  `(coq lang 0.8)` which provides the right semantics for theories
  that have been globally installed, such as those coming from opam
  (@ejgallego, @Alizter)

- Bump minimum version of the dune language for the melange syntax extension
  from 3.7 to 3.8 (ocaml/dune#7665, @jchavarri)
@hhugo hhugo deleted the inline-tests-para branch October 9, 2023 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Diffing of inline tests is not concurrent Run inline tests in parallel
4 participants