A way to shorten dependency lists: wildcards or auto-dependencies

[justinfagnani]

I think a lot of monorepos will have scripts that depend on basically all subpackages. Take lit.dev's build script:

  "wireit": {
    "build": {
      "dependencies": [
        "./packages/lit-dev-api:build",
        "./packages/lit-dev-content:build",
        "./packages/lit-dev-server:build",
        "./packages/lit-dev-tests:build",
        "./packages/lit-dev-tools-cjs:build",
        "./packages/lit-dev-tools-esm:build"
      ]
    },

This list is prone to getting out of sync as packages are added. That might be caught quickly when the project doesn't build correctly, but the list is still cumbersome.

Lerna and npm solve this by allowing you to run a script by name in any package that has that script. I could see wireit supporting something similar in a few ways:

1. Wildcards:

  "wireit": {
    "build": {
      "dependencies": [
        "./packages/**/*:build",
      ]
    },

2. Wildcard with optional script name

  "wireit": {
    "build": {
      "dependencies": [
        "./packages/**/*", // Uses "build" automatically
      ]
    },

2. Auto-deps based on "workspaces" field
   This would read the "workspaces" field as a single source of truth for the list of all subpackages and run the build script in each:

  "wireit": {
    "build": {
      "dependencies": "auto"
    },

This version of the idea would really only be useful on the top-level package. To work in workspaces automatically would require some magic around looking at which dependencies are local to the monorepo and running scripts in them. Maybe that's doable.

Since packages would still need their scripts and dependencies specified, even if auto, it seems like this wouldn't require wireit to topo-sort the packages.

[aomarks]

#23 covers adding a $WORKSPACES:<script> dependency syntax, which would mean "run <script> in all of this package's workspaces".

That's basically equivalent to npm run <script> -ws, except that we can run the scripts in parallel, instead of in serial.

I do have a feeling some of the "I need to build everything" use cases might actually come up less frequently than we are used to in Wireit projects, because the dependencies are now more explicit. E.g. we are used to typing "npm run build" in the root of our monorepo before we do anything, but you probably don't want to do that anymore. You would instead just jump straight to the package you want to work on, type npm test watch or something, and anything that needed to build will build.

A top-level test script seems like it will still be common, though, so that's where I imagine this being used.

[justinfagnani]

I was thinking top-level build would be good for CI, but with caching even that might not be necessary. I could see it being used to check things after updating dependencies, but npm run build -ws might take care of that.

[luxaritas]

Nx is able to automatically determine package dependencies within a monorepo, so building one package builds all of its dependencies without having to specify what those dependencies are beyond just adding them via a package.json (or direct import in their default setup which doesn't use workspaces, but that's not really relevant here). Automatically inferring dependencies from a package.json in a workspace seems desirable so that you don't need to specify those dependencies (effectively) twice.

[luxaritas]

Another thing Nx does is let you specify default dependencies across all workspaces - ie, in a root configuration file you can note that for the build target in all workspaces, the build target of any dependencies within the package also need to be run

[rictic]

Automatic package-level dependencies are convenient, but there's tradeoffs there too. Script level dependencies can be more granular. e.g. suppose ./foo:build depends on ./bar:build but ./foo:test doesn't depend on ./bar:test. If a build system can't represent that, then it must invalidate caches more often and run with less parallelism.

[luxaritas]

That's where configurability comes in. As I said, what Nx does is let you specify default dependencies across all workspaces for a given script. So you can say the build script in a workspace always depends on the build scripts of its dependencies - that doesn't affect the dependencies of the test scripts. I think it is relatively common for this configuration to work without issues - I'm hard pressed to imagine a situation where a build of a package would not require the dependencies being built. But even if there is some such situation, I'm sure there's other ways to make the configuration more flexible (or at worst, just don't use automatic dependency discovery)

[calebdwilliams]

I'm testing wireit for a monorepo at Capital One and will say having automatic dependency checking and script utilities are going to be a huge factor for us in determining our monorepo management tool.

If wireit could replicate a feature like lerna run in some capacity it would be a huge benefit. Workspaces in npm and yarn are close enough but the ability to automatically ignore packages in which that script doesn't exist is crucial for our use case.

I'd be happy with any of the proposals @justinfagnani put forward in the original post, but numbers one and three (really the second "2") make the most sense to me. Big fan of this feature.

[aomarks]

If wireit could replicate a feature like lerna run in some capacity it would be a huge benefit. Workspaces in npm and yarn are close enough but the ability to automatically ignore packages in which that script doesn't exist is crucial for our use case.

You can ignore scripts that don't exist using npm workspaces like this:

npm run build -ws --if-present

https://docs.npmjs.com/cli/v7/using-npm/workspaces#ignoring-missing-scripts

If wireit could replicate a feature like lerna run in some capacity it would be a huge benefit. Workspaces in npm and yarn are close enough but the ability to automatically ignore packages in which that script doesn't exist is crucial for our use case.

See also #23, which would provide an explicit way to say "run <script> in all of my workspaces".

Do either or both of those sound like they would work for your use case, @calebdwilliams ?

[calebdwilliams]

@aomarks that would work assuming it follows the --if-present logic (side note, thanks for pointing that argument out to me, I've been searching for that behavior just haven't found it).

If we can somehow opt in to caching across all packages inside the workspace that would be ideal. And if we could have a dependencies keyword that opted in to assumed script dependencies based on the package dependencies, that would be 🔥.

[gunta]

I've moved from npm-run-all to wireit for paralllel scripts processing and Glob-like pattern matching for script names was the first thing I missed out.

https://github.com/mysticatea/npm-run-all/blob/master/docs/run-p.md#glob-like-pattern-matching-for-script-names

[aomarks]

I've moved from npm-run-all to wireit for paralllel scripts processing and Glob-like pattern matching for script names was the first thing I missed out.

https://github.com/mysticatea/npm-run-all/blob/master/docs/run-p.md#glob-like-pattern-matching-for-script-names

Could you provide an example of where you would like to use this feature? I am curious about whether #23 (support $WORKSPACES syntax) would also solve this problem for you -- which would be the case where you want to run the same script name in all of your workspaces.

[aomarks]

Closing in favor of #23