Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build workspace packages in top-sorted order #1018

Merged
merged 66 commits into from
Oct 5, 2023
Merged

Conversation

JordanMartinez
Copy link
Contributor

@JordanMartinez JordanMartinez commented Sep 26, 2023

Description of the change

First part of #988.

  • When building all workspace packages, builds them in a reverse-topological-sort order (i.e. leaves first, then roots)
  • Adds a duplicate module check when building all workspace packages

Tests added cover these cases:

flowchart TD
  subgraph "Case 1"
    A ---> Dep0["effect, console, prelude"]
    B ---> Dep0
  end
  
  subgraph "Case 2"
    A2 ---> effect2
    A2 ---> console2
    A2 ---> prelude2
    A2 ---> Shared2
    B2 ---> effect2
    B2 ---> console2
    B2 ---> prelude2
    B2 ---> Shared2
    Shared2 ---> prelude2
  end

  subgraph "Case 3"
    A3 ---> effect3
    A3 ---> console3
    A3 ---> prelude3
    A3 ---> B3
    A3 ---> C3
    B3 ---> effect3
    B3 ---> console3
    B3 ---> prelude3
    B3 ---> C3
    C3 ---> prelude3
  end

  subgraph "Case 4 (duplicate module)"
    A4 ---> effect4
    A4 ---> console4
    A4 ---> prelude4
    B4 ---> effect4
    B4 ---> console4
    B4 ---> prelude4
    C4 ---> prelude4
  end
Loading

Checklist:

  • Added the change to the "Unreleased" section of the changelog
  • Added some example of the new feature to the README
  • Added a test for the contribution (if applicable)

P.S.: the above checks are not compulsory to get a change merged, so you may skip them. However, taking care of them will result in less work for the maintainers and will be much appreciated 😊

@JordanMartinez
Copy link
Contributor Author

The three cases I test for are similar to but not quite those described in the #988. Unless we publish a package that intentionally has a warning in it, it will not be possible to test for those cases specifically. So, the best we can do is test against local workspace packages that satisfy that criteria.

Future changes will cause Fetch, Repl, and Build to have such a cycle
- Fetch.run returns back the same value as it
did before (i.e. `dependencies`) but now under
the `dependencies` field name.
Under the `packageDependencies` field name,
it returns a map that can be used to look up
all the dependencies for a given workspace package.
This is used in Build.run to get all the dependencies
for a given package. However, to produce the `packageDependencies` value,
we have to call `getTransitiveDeps` multiple times
and don't get to share the work done in previous calls.
So, if 5 packages all depend on `halogen`, we're recomputing
`halogen`'s transitive dependencies 5 times.

- Fetch.run's internal implementation was refactored a bit
to prevent the impossible case where `workspace.selected` is `Just`
but calling what was `getConfigPath` was `Nothing`.
@JordanMartinez
Copy link
Contributor Author

JordanMartinez commented Sep 27, 2023

Tasks remaining:

  • Check for duplicate modules in the workspace packages and error if found
  • Add a test for the above check

@JordanMartinez
Copy link
Contributor Author

@f-f For this code

-- Every package can be imported by several project modules.
-- In each project module, several modules from the same package can be imported.
type ImportedPackages = Map PackageName (Map ModuleName (Set ModuleName))

could you clarify what the mappings are?

@f-f
Copy link
Member

f-f commented Sep 28, 2023

The comment about the type is meant to clarify just that, but I guess it can be cryptic - the idea of the mapping is: for every Package that we depend on, we note which Module we are depending on, and for each of them we note from which Module we are importing it.

@JordanMartinez
Copy link
Contributor Author

We'd return this list of errors from the Psa command instead of killing the process there, and draw the final verdict on the build only once they all succeed or one of them fails.

How does this work when a non-js backend is used? There's already a "Backend build succeeded" message.

@JordanMartinez
Copy link
Contributor Author

Remaining tasks:

  • the ensureRanges test I have yet to add
  • address feedback on latest changes
  • address the psa Build Succeeded issue

@f-f
Copy link
Member

f-f commented Oct 4, 2023

How does this work when a non-js backend is used? There's already a "Backend build succeeded" message.

Ah yes, indeed - when building multiple packages with a backend we'd first build with purs, then build with the backend, then repeat. Alright, let's leave this out of this PR and just open an issue once we merge

Copy link
Member

@f-f f-f left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great now - thanks @JordanMartinez!

Let's add the last test for ensureRanges, then we can merge in

@JordanMartinez
Copy link
Contributor Author

In latest commits, I...

  • moved the build-in-topological-order tests into their own describe block
  • kept case 4 (duplicate module check) in the same location, but renamed it for clarity
  • moved Mermaid diagrams above each test for readability, and linked to the guide for how to write one for future users
  • added --ensure-ranges tests:
    • poly repo where a root package does and does not exist
    • monorepo where a root package exists. I didn't add one for when a root package does not exist as I don't think this will ever happen.

I won't merge this despite the approval in case you have more feedback. But if it looks good, then feel free to merge it.

@JordanMartinez
Copy link
Contributor Author

How does this work when a non-js backend is used? There's already a "Backend build succeeded" message.

Ah yes, indeed - when building multiple packages with a backend we'd first build with purs, then build with the backend, then repeat. Alright, let's leave this out of this PR and just open an issue once we merge

I've tracked our conversation about this here and added it to the original issue: #988 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants