[DO NOT MERGE] Configure esy monorepo workflow (2)

[andreypopp]

This is a 2nd attempt at esy monorepo workflow (replaces #42).

The PR works only with esy nightly builds for now:

% npm install -g @esy-nightly/esy

How it works

esy.json links to each package in the repository using
link-dev:./path/to/package.json.

Each package linked with link-dev: behaves exactly like a root package in a regular esy workflow:

• Its corresponding "devDependencies" are installed
• "buildDev" command is executed on build (if it's defined, falls back to "build")
• Build commands have access to their corresponding "devDependencies"

Workflow

Install dependencies:

% esy install

Build dependencies (each package has access to its "devDependencies", runs
"buildDev" command if it's defined):

% esy build

Build dependencies in release mode (run "build" command w/o access to "devDependencies"):

% esy build --release

Discussion

Is it possible to run builds commands for a specified package?

Yes, use esy build-package command:

% esy build-package ./rely.json

Is it possible to run commands in a build env of a specified package?

Yes, use esy build-exec command:

% esy build-exec ./rely.json ls

The command esy build-exec CMD is an analogue of esy build CMD command but
allows to specify the package.

Is it possible to make change detection more granular?

When one changes sources (do touch ./src/console/Console.re for example) all
packages are being rebuilt (actually they are not being rebuilt, esy just runs
build commands but as dune works incrementally those commands are noop).

It's possible to make esy detect changes in a more granular fashin but that requires moving package roots into different dirs, for example:

% mv rely.json src/rely/package.json

I've experimented with this approach and it doesn't work with the current setup
for common - if we move package root to the src/ then copy_files doesn't
work as it can't copy files from outside the package root.

If we change how we vendor code from common/ dir we can enable more granular
change detection with esy.

Changes

• 394ecfc

  Convert to esy monorepo

  • esy.json links to all packages via link-dev: This collects & installs
    all "devDependencies" for packages and makes them available to 'build"
    command (or "buildDev" if it's configured).

  • Fixes src/refmterr/lib/dune to have access to Common (not sure why it's
    worked before).

  • Adds "buildDev" with refmterr for those packages
    (file-context-printer) which only rely on refmterr in dev.

• c53ee3f

  Add @reason-native/tests package

  This package hosts tests.

• 1d24018

  Use esydune wrapper for dune

  This configures "build" (and "buildDev" where needed) with esydune
  instead of dune. This makes sure we invoke dune with the right arguments,
  this effectively enables warnings in reason-native.

  Then this commits update dune files with flags to disable erroring on 29
  (unused bindings) and 9 (unbound record fields in patterns) warnings.

• 5310521 Update esy.lock (not for review)

[jordwalke]

Should we make esy's built in install process automatically look to see if there's a .install file that matches your package name (minus any scope) and just use that one so you don't have to specify it or use esydune install?

[andreypopp]

Good idea! Will implement now.

[andreypopp]

PR updated not to include redundant "install" config.

[jordwalke]

% esy build --release

That seems good to have even without monorepos.

Is it possible to run builds commands for a specified package?
Yes, use esy build-package command:

% esy build-package ./rely.json
Is it possible to run commands in a build env of a specified package?
Yes, use esy build-exec command:

% esy build-exec ./rely.json ls
The command esy build-exec CMD is an analogue of esy build CMD command but
allows to specify the package.

I think these could be confusing to newcomers just because they already have to
learn esy @pkg.json cmd. It's not immediately clear to noobs how it's
different. Also esy build-exec sounds like esy x ("esy exec").

What about:

esy build --package=./rely.json cmd

Then it's clear this is just an extension of the esy build cmd command - not
something really new to learn.

This is a kind of rare command to need, so a little extra verbosity would not be bad.

It's possible to make esy detect changes in a more granular fashin but that
requires moving package roots into different dirs, for example:
% mv rely.json src/rely/package.json

I'm confused because I thought that for linked packages with out of source
builds, esy won't do anything to try to do change detection, it will just call
the build system for each linked package and expect it to be incremental. What
would we gain by doing some more granular sub-package change detection for
linked packages if we never use that change detection for linked packages?

Other question: How was the circularity solved?

[andreypopp]

% esy build --release

That seems good to have even without monorepos.

Yep, this is needed as we've added buildDev.

Is it possible to run builds commands for a specified package?
Yes, use esy build-package command:

% esy build-package ./rely.json
Is it possible to run commands in a build env of a specified package?
Yes, use esy build-exec command:

% esy build-exec ./rely.json ls
The command esy build-exec CMD is an analogue of esy build CMD command but
allows to specify the package.

I think these could be confusing to newcomers just because they already have to
learn esy @pkg.json cmd. It's not immediately clear to noobs how it's
different. Also esy build-exec sounds like esy x ("esy exec").

What about:

esy build --package=./rely.json cmd

Then it's clear this is just an extension of the esy build cmd command - not
something really new to learn.

This is a kind of rare command to need, so a little extra verbosity would not be bad.

It's possible to make esy detect changes in a more granular fashin but that
requires moving package roots into different dirs, for example:
% mv rely.json src/rely/package.json

This can be done for esy build CMD but:

• cmdliner doesn't expose the API for that and thus I have a hardcoded hack
  for --release arg, we can add the same hack for --package PKG too.
• Other commands like esy build-plan / build-env / ... already accept package
  as position argument. Though I suppose we can change that.

Overall - I agree, let's implement it that way instead of adding a new command.

Other question: How was the circularity solved?

Packages now specify their dependencies/devDependencies in their own
package.json files and thus they control circularity with needed granularity:

• Those packages which refmterr depend on can't specify refmterr as devDep
• Others can

[andreypopp]

I'm confused because I thought that for linked packages with out of source
builds, esy won't do anything to try to do change detection, it will just call
the build system for each linked package and expect it to be incremental. What
would we gain by doing some more granular sub-package change detection for
linked packages if we never use that change detection for linked packages?

Nope, we always check for staleness but I think we could indeed don't do that for out of source builds.

[andreypopp]

New CLI in latest nightly allows to use esy build CMD with -p/--package option to target specific packages, for example:

% esy build -p ./rely.json dune runtest

Therefore esy build-exec is no longer needed.

[jordwalke]

I'm confused because I thought that for linked packages with out of source
builds, esy won't do anything to try to do change detection, it will just call
the build system for each linked package and expect it to be incremental. What
would we gain by doing some more granular sub-package change detection for
linked packages if we never use that change detection for linked packages?

Nope, we always check for staleness but I think we could indeed don't do that for out of source builds.

Okay, so if that optimization is made for linked packages then we can continue to keep the package config .jsons at the root without any penalty it seems. (People could move them down deeper if they like, but it's nice that they wouldn't have to).

[jordwalke]

Looking really good!
I think we really need an esy version field in the json asap. I wish we had one already supported earlier because then we could just land this, and users on older versions of esy would get a nice error message saying they need to do npm remove -g esy && npm install -g esy

[andreypopp]

Update: dune has merged PR (ocaml/dune#1750) which enables merlin to see original source locations for linked packages in a monorepo. I'm adding needed fixes for esy to enable that. After dune new version is released this PR could be merged.

[kyldvs]

I'm guessing this is no longer relevant and can be closed given how old it is.