install subcommand design

[bobot]

The current state of the install subcommand is currently
unsatisfactory (cf. #250, #372, #202). I promised to improve it, but I
have not delivered the code due to lack of time, and the time needed for understanding the current usage. I perhaps found someone willing to do it, but the design must be set
first. I'm going to edit this comment with the current state of the
design once something is agreed on.

[bobot]

The main goals are:

1. to be able to support opam environnement but also usual
   distribution. The place where distribution put files is similar to
   opam (standard filesystem hierarchy)
   except for /etc and for the ocaml library files.
2. to support the DESTDIR feature out-of-the-box.
3. to keep all the section from opam install file, except misc and with
   the addition of ocamllib:
   ocamllib, lib, libexec, bin, sbin, toplevel, share, share_root ,
   etc, doc, stublibs, man. Supporting misc means to be able
   to ask the user if it accept to put files in an unorthodox location and
   to understand what is the location given by the developper in the
   different environments. Adding ocamllib is needed because linux
   distribution put it in /usr/lib/ocaml and not $PREFIX/lib like opam.

So the design is:

• install subcommand should work in different layout namely (in
  order of precedence):

  • opam: installed from opam (configure.ml --opam-prefix prefix)
  • base: use autoconf defaults and options (e.g. /usr/local and
    --prefix), with the help of ocamlfind for ocaml path

• configure of dune (configure.ml) allows to select the
  environment:

  • --opam-prefix $prefix select opam layout at directory
    $prefix, so for installing package $pkg:
    • ocamllib, lib, libexec: $prefix/lib/$pkg
    • bin: $prefix/bin
    • sbin: $prefix/sbin
    • toplevel: $prefix/sbin
    • share: $prefix/share/$pkg
    • share_root: $prefix/share
    • etc: $prefix/etc
    • doc: $prefix/doc/$pkg
    • stublibs: $prefix/lib/stublibs
    • man: $prefix/man
  • if --opam-prefix is not used the base layout is used and the following options could be
    used (from configure generated by autoconf):
    • --prefix=PREFIX install architecture-independent files in PREFIX [/usr/local]
    • --bindir=DIR user executables [EPREFIX/bin]
    • --sbindir=DIR system admin executables [EPREFIX/sbin]
    • --libexecdir=DIR program executables [EPREFIX/libexec]
    • --sysconfdir=DIR read-only single-machine data [PREFIX/etc]
    • --libdir=DIR object code libraries [EPREFIX/lib]
    • --datarootdir=DIR read-only arch.-independent data root [PREFIX/share]
    • --datadir=DIR read-only architecture-independent data [DATAROOTDIR]
    • --mandir=DIR man documentation [DATAROOTDIR/man]
    • --docdir=DIR documentation root [DATAROOTDIR/doc/PACKAGE]
      For ocamllib: --ocamllib=DIR ocaml library root [LIBDIR/ocaml/]
      or the result of ocamlfind printconf destdir is available.
      So for installing package $pkg the layout is:
    • ocamllib: $OCAMLLIB/$pkg
    • lib : $LIBDIR/$pkg
    • libexec: $LIBDIR/$pkg
    • bin: $BINDIR
    • sbin: $SBINDIR
    • toplevel: ??
    • share: $DATADIR/$pkg
    • share_root: $DATAROOTDIR
    • etc: $SYSCONFDIR
    • doc: $DOCDIR
    • stublibs: $OCAMLLIB/stublibs
    • man: $MANDIR

• configure of a package dune configure allows to override the
  layout with the same options. The choice is written in an sexp file
  in _build.

• The following options of configure from autoconf are not kept
  --exec-prefix, --sharedstatedir, --localstatedir,
  --runstatedir, --includedir,
  --oldincludedir, --infodir, --localedir, --htmldir,
  --dvidir, --pdfdir, --psdir. But --includedir could be useful.

• When copying file to install to _build, only one layout must be
  used for reproductibility. And I think the base layout (with
  default directory) is preferable because the sections are more
  separated (ocamllib,lib,libexec).

• Variable for all the final locations of the sections must be accessible in
  jbuild configuration file (e.g. $bindir). Also the local version
  (in _build) must be accessible once Configure variable set by jbuilder exec #205 is done.

• Same handling of .exe as currently.

• Should windows be supported only with opam? Otherwise what are a
  good default?

@Andily (for debian), @AltGr (for opam), @dra27 (for windows) what do
you think of that?

[ghost]

/cc @nojb @alainfrisch as well for the Windows part. I remember we talked about jbuilder and installation on Windows in the past

[frispete]

Thanks @dra27 for pointing me to this place.

I would like to give you some real world inspiration to the problem.

Here here two projects, which were ported to dune lately:
https://build.opensuse.org/package/show/home:frispete:ocaml/ocaml-gapi
https://build.opensuse.org/package/show/home:frispete:ocaml/google-drive-ocamlfuse

The spec files define typical rpm package builds.

An attempt to use jbuilder install failed completely due to current limitations. I came up with:

%build
jbuilder build @install

%install
opam-installer \
    --prefix=%{buildroot}%{_prefix} \
    --docdir=%{buildroot}%{_docdir} \
    --libdir=%{buildroot}%{_libdir}/ocaml \
    --mandir=%{buildroot}%{_mandir} \
    %{origname}.install

# remove docs: they landed in %{_docdir}/%{origname}
rm -rf %{buildroot}%{_docdir}

which isn't as straight forward as one would wish for... Probably I've missed some opportunities to optimize this, but in order to make this more manageable, the DESTDIR feature is crucial, and creating any missing directories another.

[bobot]

Thanks @frispete for this post-mortem analysis and the pointers. Do you think the current proposition will work for rpm package?

[frispete]

As a packager, the DESTDIR feature is most important. As you can see from the snippet above, all used paths have to be specified in order to get opam-installer to install into another destination. Hence, a packager needs to determine, which paths are needed.

Most useful would be an install command like:

%install
DESTDIR=%{buildroot} jbuilder install

or variations thereof.

The rest of the proposition looks reasonable from my POV, but beware, I'm not that deep into the ocaml specific installation details, sorry.

[rgrinberg]

I'm closing this ticket as diml already implemented most of this. The only thing that remains is the DESTDIR env var which already has its own ticket.

[ghost]

@rgrinberg I believe that the scope of this feature is broader than what I did + support for DESTDIR. For instance it allows to support more complicated installation layout either by configuring dune differently or via a new dune configure command

[rgrinberg]

Okay. It would still be helpful to spec out dune configure separately I think however. It doesn't look as nice when open proposals are partially implemented.

[bobot]

Hi @glondu, since you packaged dune for debian do you have any comment on this proposition which could touch the way packages that use dune are packaged?

[ArchangeGabriel]

The proposal is not even fully respected currently, e.g:

--datarootdir=DIR read-only arch.-independent data root [PREFIX/share]
--datadir=DIR read-only architecture-independent data [DATAROOTDIR]
--mandir=DIR man documentation [DATAROOTDIR/man]
--docdir=DIR documentation root [DATAROOTDIR/doc/PACKAGE]

DATAROOTDIR is apparently set to PREFIX only, and I still have to fix it on my side.

But we have great progress now with DESTDIR being supported. :)

[ejgallego]

It seems to me that most of this is solved by various advances in the codebase, I propose we close this issue and keep #3978 open for now.

[rgrinberg]

Yes, this ticket is quite confusing and I'm not sure which parts are still relevant. I also vote to close and create a new issue with the relevant points

[ejgallego]

Yes, this ticket is quite confusing and I'm not sure which parts are still relevant. I also vote to close and create a new issue with the relevant points

Actually I was wrong on my analysis, this issue is still pretty much relevant, as of today some of the ideas here are needed, otherwise dune install can't work for Debian systems, see for example #4723

[ejgallego]

Note that we committed a quick fix for 2.9 in #4744 , by adding support for etcdir and docdir, this helps quite a bit as it solves (I believe) all issues with Debian packages, but there was a TODO in such PR that was left for 3.0:

How about we just treat all directories the same way and make them all overridable? e.g. share. I don't see the downsides of doing this, and the upside is that it's one less thing to remember for users. It's rather annoying to maintain a map in your head between overridable directories and versions of dune.

This wasn't done as in the PR as backporting to 2.9 would have been tricky.

[rgrinberg]

We've had more recent changes in this department so it's worth revisiting where we stand with the rest of the team. @bobot could you let us know where things stand at the next meeting?

[kant2002]

If anybody would have a problem with bringing up dune on the system where there no exept working compiler (like loongarch64), here the commands which allow install dune globaly.

PREFIX=/usr/local ./configure --mandir $PREFIX/man --docdir $PREFIX/doc/ocaml --etcdir $prefix/etc --datadir  $PREFIX/share --bindir $PREFIX/bin --sbindir $PREFIX/sbin --libexecdir $PREFIX/libexec --libdir  $PREFIX/lib
make release
make install

I can submit PR to README, but if this would be implemented, that's would be no-op, since command lines would work.

[maiste]

This issue doesn't seem to get more traction anymore. According to the comments above and the changes made, we consider it as done. Feel free to reopen if you think this is relevant 👍