Overhaul include/preamble semantics (and lots of plumbing for the instantiation / run cycle)

[alerque]

Closes #653
Closes #798
Closes #1092
Closes #1413
Lays groundwork for #1438

---

BREAKING CHANGE: The -I / --include option whose use was overloaded for more than one purpose is deprecated in favor of more specific replacements: -r / --require for loading code into SILE before input processing, -p / --preamble for processing content prior to a document and -P / --postamble for processing content after a document.

c.f. especially my latest romment

Having reviewed this a bit further, I don't think the -I option is even fixable. It tries to be too smart and as a result can't easily be manipulated to do any specific thing.

• Does it load Lua code? or process SILE content?
• Does it run before loading the parser or after?
• Does it search for files relative to the PWD? Or the input document? Or the Lua search path?

Some of these combinations are mutually exclusive and guessing can lead to breakable. I propose deprecating it entirely in favor of more specific and predictable alternatives.

[alerque]

I haven't actually implemented this change yet, just updated the CLI hints and man page for what I am proposing. This draft PR is a request for comment, especially from @simoncozens.

Here is what the new bits of man page would be:

       -I, --include=filename
              Deprecated, will be removed.  Please use --require, --preamble, or --postamble.

       -r, --require=module
              Require a class, package, or other code to be loaded before processing the main
              input.  The value should be a loadable module name without a file extension and
              will be loaded using SILE's module search path.  This is particularly useful if
              the input file is in a non-native XML or other format and  requires  loading  a
              class or other code that extends SILE to understand how to read and process the
              input document.  May be specified more than once.

       -p, --preamble=filename
              Include a SILE, XML, or other content file  before  the  input  document.   The
              value  should  be  a  full  filename with a path relative to PWD or an absolute
              path.  May be specified more than once.

       -P, --postamble=filename
              Include a SILE, XML, or other content file after the input document.  The value
              should be a full filename with a path relative to PWD or an absolute path.  May
              be specified more than once.

[alerque]

A couple more notes:

• We could expand the -r option to be able to read SIL files using the Lua search path (and hence finding things in the SILE installation rather than the current project) but I don't really see the point. We can rewrite the docbook class in Lua, and I haven't come up with other use cases for this that really make sense. Yet.

• We currently have something internally called "preamble", but it isn't actually handled as a preamble. I would suggest the actual --preamble content be handled after processing the document declaration from the main input AST, but before diving any farther into the AST. This seems like it would be more utilitarian than doing anything with it before we even know the papersize or class.

[Omikhleia]

We currently have something internally called "preamble", but it isn't actually handled as a preamble. I would suggest the actual --preamble content be handled after processing the document declaration from the main input AST, but before diving any farther into the AST. This seems like it would be more utilitarian than doing anything with it before we even know the papersize or class.

Erm, maybe I did not fully understand the point, but for many XML cases, I feel that the current "preamble" is actually what is needed: a document that will define the papersize, class and other settings or options, before processing the XML content.
For non-XML cases, this would still be valid, e.g. by using a "preamble" before a document, I can have a generic one, say, for book-size paper, high resolution, etc. and another for other purposes (A4 size, other options), as an easy generic switch.

EDIT: Or maybe the above is covered by your -r now?

[alerque]

I think (still playing with exactly which option does what) you would still be able to use a pre-document-document roughly like you are now, but with the new arguments I would envision you loading up a class using sile -r classes.mything document.xml. I actually have a POC over here with that working, but I'm not quite sure where class options should go any more. It is redundant to also send a preamble to set the class, but it does make it more obvious how to set options. It also crossed my mind we could do it with a more specific semantic such as sile --class mything --options papersize=a6 document.xml.

[Omikhleia]

Ah I see the idea.

In the cases I consider, it could perhaps be the same class, with different class options, but also:

• other commands (possibly via additional packages, e.g. my "print" preamble could load my experimental printoptions package to fix image resolution and rasterize vectors; while my "preprint" version could do none of this, but load cropmarks and set the target paper to A4, etc.
• other settings (e.g. a "review" version could be on full A4 with larger font and huge base line skips for annotations)

For an actual use case, e.g. my XML lexicon is multi-lingual, my current preambles selects the main language and related options (e.g. the English version skips all French elements, the French version skips most of the English but keeps some of it such as glosses, since the definition come (more or less) from the English author. Currently it the same class, and the preambles (one for "fr", one for "en") just enables different settings for the specific output. Of course in that case, I could have maybe used class options rather than settings, but that could make a lot of them... And having to remember all those to pass as command line argument is also a bit unsatisfying.

That is to say, I am unsure requiring a class only and playing with CLI options is the way to go. But I am not certain the above makes full sense either!

[alerque]

I don't see any reason you couldn't mix and match. What you are doing is fine if that's what you want and I don't see any reason to change it. The issue I have is that there are things the current system will not let you accomplish. It also has weird gotchas like not being able to include SIL things that happen to have the same name as a Lua thing (or vise versa). There just isn't enough control currently.

[alerque]

I realize there is more work to be done along these lines but testing and reasoning about all the changes in context with other developments was starting to get too hard in a branch.