texlive: switch to buildEnv/withPackages

[xworld21]

This is to track various changes to modernize texlive and make it work like haskell, lua, perl, python, which seem to have converged on the same interface. This comes after quite a lot of work to make the existing texlive almost 100% functional and with lots of tests, so we should be able to spot breakages easily. That's the theory anyway.

Feedback and comments are very welcome. Cc @veprbl @apfelkuchen6.

Intended outcome

Building a new TeX Live environment:

texlive.withPackages (ps: with ps; [ scheme-small dvipng ])

Building a TeX Live environment with documentation or a different paper size:

# replace arguments of buildEnv
texlive.buildEnv.override { packages = ps: with ps; [ scheme-small ]; extraOutputsToInstall = [ "texdoc" ]; }
# extend arguments of buildEnv (e.g. add packages)
texlive.scheme-small.buildEnv { extraOutputsToInstall = [ "texdoc" ]; paperSize = "letter"; }

Building a TeX Live package outside of texlive

stdenv.mkDerivation { outputs = [ "out" "tex" ]; ... }

Replace a TeX Live package with a different version, or pass other overrides

texlive.scheme-small.override { packageOverrides = final: prev: { asymptote = asymptote; }; }

The changes will also make it easier to ship different snapshots of TeX Live, or at least to upgrade/downgrade locally, and to package tlcontrib. We should also become able to deduplicate the formats.

Roadmap

Here is a tentative roadmap. I believe we have enough of the difficult parts in draft form to make it by 23.11, and be prepared to drop the old stuff in 24.05/24.11, so that we make a single big change. We should really try to do this once: I already made a mistake this cycle when introducing tlDeps, which in hindsight was the wrong choice (see requiredTeXPackages below)!

Still, each step should be fully backward compatible, so we can stop halfway if we don't make it.

Internals (no rebuild, fully backward compatible, new interface undocumented and prefixed with _)

0. Procure more tests! tests.texlive: several new tests #248960 for start, plus texlive: fix tlmgr, tlshell, licenses #247798 to fix tests.texlive.licenses.
1. Refactor the packages outside of texlive (fix texlive.combine: package 'combine' from CTAN not usable #237227). Edit: new scope postponed as it takes more thinking than I expected.
   • Refactor package building in separate buildTeXLivePackage (ready in texlive: refactor package builder in separate expression #247550). Strictly for TeX Live packages so tightly coupled with tlpdb.nix, not for general use.
2. Use a 'multi-output' approach for TeX Live packages. Proof-of-concept in [WIP] texlive: change TeX Live packages into attribute sets instead of lists #249803: not literally multi-output, packages are still groups of fixed output derivations that build independently of each other.
   • Rewrite buildTeXLivePackage accordingly (ready in texlive: multi-output packages (subset of #250805) #250963).
   • Backward compatibility: provide 'alias' texlivePackages.example.pkgs with old tlType interface, set texlive.example = { inherit (texlivePackages.example) pkgs; }. Edit: maybe there won't be a texlivePackages.
3. Implement texlive._buildEnv and shorthand texlive._withPackages.
   • Rewrite combine logic for multi-output packages (mostly ready in texlive: overrideTeXConfig/withPackages #250805; proof-of-concept with more improvements in [WIP] texlive: change TeX Live packages into attribute sets instead of lists #249803). Suggested arguments: { extraName, extraOutputsToInstall, extraVersion, packageOverrides, packages, paperSize }. For instance extraOutputsToInstall = [ "texdoc" ] adds the package docs; packages = [ example example.texdoc ] adds the docs for example only. (paperSize implemented in texlive.combine: add paperSize parameter for setting paper sizes #248768).
   • Change tlDeps = [ ... ] dependencies into requiredTeXPackages = ps: [ ... ]. Otherwise the packages may be pulled from the wrong scope.
   • Backward compatibility: keep accepting tlDeps for dependencies and pkgFilter for adding or removing packages (ready in texlive: overrideTeXConfig/withPackages #250805).
   • Make combine a thin layer over buildEnv (ready in texlive: overrideTeXConfig/withPackages #250805).

Switch on new interface

4. Enable .withPackages.
   • Switch Nixpkgs to .withPackages treewide: switch to texlive.withPackages #265514.
   • Document .withPackages (texlive: document new texlive.withPackages interface #265658).
   • Stabilize .overrideTeXConfig and document it.
   • Add comments to Release Notes.
   • Document new way of building TeX Live packages (as 'multi-output' packages) and deprecate old tlType, tlDeps, pkgs.
   • Expose TeX Live packages in search? Which also means being honest about how many packages are rebuilt during upgrades.

Next release cycle?

5. Make the TeX Live packages overrideable.
   • Decide a stable interface for buildTeXLivePackage, decoupled from the quirks of tlpdb.nix as much as possible.
   • Expose overrideTeXLiveAttrs.
6. Create a properly spliced scope for texlive; it must be possible to override binaries and packages simultaneously in a consistent way, and supporting cross compiling as far as possible (buildEnv cannot be cross compiled, unfortunately). Discussion in progress at texlive: create scope #250416 and texlive: transform into overridable scope and extract generic interface #250626.
7. Deprecate .combine.

By deprecation, I mean add a deprecation notice with lib.warn and use config.allowedAliases to prevent usage within Nixpkgs.

Ideal scenario, we merge everything by the end of September, to have a good period of testing before 23.11.