Optional argument for --promote-install-files breaks CLI compatibility

[dra27]

Dune 2.9.0 adds an optional argument to the --promote-install-files flag. This breaks compatibility with previous command lines, as it means any parameter following the parameter-less version of --promote-install-files for 2.8.x and earlier is now interpreted as the argument to --promote-install-files.

This has hurt opam, which post-processes its .install files and so relies on promotion (cf. ocaml/opam#4752 and ocaml/opam#4753). We've worked around it by putting --promote-install-files last, but that's only an option as long as Dune does this once!

From the opam side, it's slightly amusing as we just put a huge development effort into --cli versioning, but I don't think that Dune needs that! Would it be possible, given that 2.9.1 is on the horizon, to implement this as --promote-install-files and --no-promote-install-files?

The odd thing in this instance is being forced to upgrade Dune in order not to use a new feature, so it sort of feels like a regression?

[bobot]

Funnily the first implementation used --no-promote-install-files (#4730 (comment)), but that was before we dived into those pesky optional arguments.

I would prefer to understand and fix those optional arguments (they should only accept --opt=v), but if we can't in the 2.9.1 timeframe, doing what you propose is very fast.

[dra27]

In terms of cmdliner, it's behaving correctly.

Thinking further, my comment that it's a problem if Dune did this more than once is incorrect - I fixed it in opam by putting --promote-install-files last, but I could also have achieved it with a --.

[bobot]

In terms of cmdliner, it's behaving correctly.

Yes but I think it is very problematic for cli versioning since you can't use optional argument for adding argument to an option. Do you have a pointer how you want to handle cli versioning in opam? We though of looking at the dune-project version but the need to find and read this file before parsing the command line was deemed too complicated.

[dra27]

opam does a lot of pre-processing of Sys.argv (and already did before CLI versioning, because of plugins). So we pre-process the command-line looking for --cli (see OpamCliMain.filter_cli_arg).

That knowledge is then used to customise the cmdliner argument parser, so every flag, parameter and so forth gets tagged with a validity (very similar idea to fields and variables in dune files). See OpamArg.ml and OpamArgTools.ml.

I think full CLI versioning in minor releases would be overkill for Dune, though - major releases could choose to drop CLI compatibility perhaps?

[bobot]

For 3.0 we could drop compatibility with 1.* but not more. Thank you for the explanation.

[ejgallego]

Hi folks, planning to do a 2.9.1 soon, what is the status of this issue?

[bobot]

Oh gosh, I forgot it before my vacation. I will have to do the fast --no-promote-install-files, I try for the end of the day. FIY @jeremiedimino

[bobot]

I think finally that 2.9.1 should not be delayed because of this issue. As stated the problem is fixed if we do that only once. I believe we will have a more backward compatible way of extending our cmdline in the futur, with #4859 or something else.