Workspaces issue progress

[bestander]

This issue will link all related discussions and RFC.

Prior art (wrappers on top of yarn and npm):

• https://github.com/lerna/lerna
• https://github.com/whitecolor/yall and https://github.com/whitecolor/yalc
• https://github.com/sindresorhus/np (nice publish experience)

RFC:

• RFC for link: :link dependencies rfcs#34, will help when workspaces reference each other
• RFC for workspaces install workspaces phase 1: aggregated install rfcs#60
• RFC for workspaces add/remove/upgrade/* Workspaces phase 2: commands to manage workspaces rfcs#61
• RFC for workspaces linking Workspaces phase 3: linking rfcs#66
• RFC for publishConfig Initial RFC for publishConfig for yarn rfcs#40
• RFC for knit command "yarn knit" -- a command to test libraries while working on them rfcs#41 - simulates publishing a dep instead of just symlinking to a folder. That may be useful for Phase 3 described in workspaces phase 1: aggregated install rfcs#60
• RFC for project: that is similar to knit idea Add 'project:' dependencies rfcs#59
• Full workspaces RFC aggregated [RFC] Yarn Workspaces rfcs#62
• RFC for publish and workspaces (phase 5) (looking for a champion)

Pull requests:

• project: proof of concept Add 'project:' dependencies #3281
• workspaces install phase 1 workspaces: first phase: install node_modules aggregated from all workspaces #3229
• workspaces add/remove commands phase 2 Workspaces phase 2: executing commands #3365
• workspaces linking each other phase 3 & 4 Workspace phase 3 & 4 #3516
• link: PR feat(type): add support for new link: dependency type #3359

Issues/discussions:

• Yarn npm_package_config equivalent #2989 - npm config base equivalent (needs clarification)
• Option to not prune on install #2919, Allow child packages to install into a single node_modules in the root package #711, Support unpublished lerna packages – ignore symlinked packages or not found failures #2863 - common node_modules, should be fixed with Phase 1

Implementation phase:

• (0.25) 1. Installing dependencies of all workspaces in the root node_modules
• (0.26) 2. Commands to manage (add/remove/upgrade) dependencies in workspaces from the parent directory
• (0.27) 3. Ability for workspaces to refer each other, e.g. jest-diff → jest-matcher-utils inside jest monorepo
• (0.27) 4. package-hoister for workspaces, e.g. when dependencies conflict and can't be installed on root level
• 5. Remove the experimental flag and turn on by default
• 6. Command to publish all workspaces in one go (looking for a champion)

[arcanis]

@bestander @cpojer I was thinking about the workspaces field. In the RFC we merged, we decided to use globbing to find all the package data (packages/**/*). I like the syntax, but won't this be an issue from a perf standpoint? It means that any cross-workspace command that we execute must first stat a lot of sub-directories, then read all the package.json files in order to extract the names. That's unless we implement some kind of cache, of course.

[bestander]

It could add up to the start time as Yarn would need to find and read multiple package.json files.
But for most cases I don't think it will be a problem, if glob find becomes slow people could be motivated to use less aggressive glob rule.
Or we could come up with a cache.

[cpojer]

I think this should be fine from a perf-perspective for now. I expect the number of shared projects to be a small number (<100) in most cases. As @bestander said, large projects can also directly embed all the paths.

[richburdon]

yarn help workspace

Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.

Returns 404

[bestander]

Workspaces is not going to be a CLI command but rather a configuration.
I am planning to post about it here https://yarnpkg.com/blog/ and have a section on different ways to link local projects here https://yarnpkg.com/en/docs/.

[bestander]

I intend to finish the post by the end of the week.

…

On Mon, Jul 17, 2017 at 8:32 PM Rich Burdon ***@***.***> wrote: Thanks very much @bestander <https://github.com/bestander> Any idea of the timeline (for the docs, and the features to land?) Very much looking forward to it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWBLUb0kiMRl8FDmscPt1SE_alD5Bks5sPCdIgaJpZM4NNLga> .

[donaldpipowitch]

Should this issue track all workspace related bugs? If yes, I created two issues --

• Can't remove package in workspace based project #3972: Can add, but not remove packages.
• ambiguous install behaviour of workspace projects with the same name as published packages #3979: Ambiguous install behaviour, if local packages clash with names of published packages.
• Unexpected install behaviour in workspace based project #3980: Unexpected behaviour?

[bestander]

Thanks, @donaldpipowitch.
I think this issue can be closed soon.
All workspaces-related bugs we tag with https://github.com/yarnpkg/yarn/labels/feature-workspaces.

BTW, thanks a lot for submitting issues with great repro steps, very much appreciated.

[donaldpipowitch]

Thank you for your work ❤

[elderfo]

Are workspaces still under the experimental flag in the RC?

[bestander]

Yes, still under experimental

…

On Mon, Jul 24, 2017 at 11:07 PM elderfo ***@***.***> wrote: Are workspaces still under the experimental flag in the RC? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3294 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBdWHSnlGt_9sz2WgKWVb4016jM6y6Rks5sRYY8gaJpZM4NNLga> .

[richburdon]

@bestander any update on your blog or instructions for using workspaces?

[bestander]

Sorry, I am waiting my colleagues to give feedback on the blogpost, a draft is here yarnpkg/website#580

[akosyakov]

I don't know. The use cases you're talking about looks to me things that should be handled by those tools rather than by the package managers. Don't forget that each rule we add to specify how we install dependencies also decreases our ability to improve it and try out new things in later releases. That's why how the hoister works is not specified: it gives us leverage to improve it without breaking contracts.

VS code and other tools follow node module resolution as defined here, looking at package.json is just one of steps. I don't see why they should be fixed.

We follow a semantic based on package.json, which I believe is common enough for it to be natively supported by most tools. If yours doesn't, then maybe it might be worth considering implementing it. Just my personal thoughts 😉

Most of the tools follow this semantic that does not allow to access dependencies of siblings packages. What is allowed by hoisting and why it does not play nicely.

[arcanis]

Most of the tools follow this semantic that does not allow to access dependencies of siblings packages. What is allowed by hoisting and why it does not play nicely.

The Node resolution algorithm has no knowledge whatsoever of the dependencies field of the package.json. How it is interpreted is left to the package manager implementations, and the common interpretation is "I promise that if you specify a dependency, you will be able to access it". Once again, we make no guarantee that you can't require a module that is not listed, and this is by design.

Hence why I say: if you want your tools to be aware of the relationships between your packages, then you have to read their dependencies fields and interpret them, you can't just rely on the node_modules directory being in a particular state. Otherwise, you will just end up losing information.

[svenefftinge]

The package.json is a means for yarn or npm to produce the node_modules folder(s). All downstream tools, transpilers and loaders should rely on the same scoping rules, that is the node_module resolution strategy. In my projects I have typescript, vscode, webpack, node. They all use the same module resolution strategy and they all 'see' stuff that they should not, because the hoisting puts too much on their scope.

[Hypnosphi]

@svenefftinge How do you think, is it OK to import a second-level dependency directly in your code? Say you have a dependency to react in your package.json, and react depends on fbjs. So should your tools warn you if you do this?

import emptyFunction from 'fbjs/lib/emptyFunction';

[svenefftinge]

Re-exported transitive dependencies are what yarn and npm do by default, so I wouldn't argue that they should not be added. That said, with the design I proposed it would be possible to optionally only link the explicit dependencies.

[BYK]

That said, with the design I proposed it would be possible to optionally only link the explicit dependencies.

Which adds another feature/option that we need to support with increased complexity, that's the point @arcanis is trying to make. It is the toolmakers' responsibility to protect against this one.

[donaldpipowitch]

Thank you every on for this great feature. I there an issue where I can track discussion/progress for "nested workspaces"? We used this previously with Lerna and it was a really nice experience. It looked similar to this:

packages/
  - buttons/
    - examples/
      - simple/
      - group/
  - forms/
    - examples/
      - simple/
      - complex/
      - validation/
...

So basically: publishable packages on the first layer and examples for every package at the second layer (with private: true).

So basically:

{
    "private": true,
    "workspaces": [
        "packages/*",
        "packages/*/examples/*"
    ]
}

[rarkins]

If anyone is interested, I've added yarn workspaces support to Renovate. It's an open source tool for keeping dependencies and lock files up-to-date which you can run self-hosted (installable via npm/yarn) or via the GitHub App I run, if your repository is on GitHub.

[donaldpipowitch]

💡 This seems to work:

{
    "private": true,
    "workspaces": [
        "packages/*",
        "packages/*/examples/*"
    ]
}

I guess "nested" workspace means multiple package.json files containing the workspaces property than? And not having nested packages inside a workspace?

[ghost]

Hi guys, I have problem with Ember: #4503 (same as #4081)

Ember thinks package is missing, but it's installed in the root dir and it doesn't search for it.

[gustaff-weldon]

@bsvipas I think this is ember-cli and at least ember-dependency-checker-related. I suggest you bring the issue to ember-cli team attention.

[bestander]

Thanks everyone, I'll mark this issue as closed.
Please go ahead and raise issues for particular cases.