amsmath.dtx: LuaTeX version of \genfrac@rule and \genfrac@choice are buggy

[RuixiZhang42]

Brief outline of the bug

The traditional TFM-based math symbol fonts (family 2) are required to have \fontdimen20 and \fontdimen21 when constructing generalized fractions with delimiters. These two font dimensions do not have correspondences in the OpenType Math Table. So, in order to support OpenType math fonts under XeLaTeX/LuaLaTeX, amsmath provides different implementations for \genfrac.

However, there seems to be illogical code in the LuaTeX version.

Minimal example showing the bug, Log file (required) and possibly PDF file

Not applicable. The use of OpenType math fonts needs the unicode-math package, which is forbidden by \RequirePackage{latexbug}. And it’s too much work to write \textfont1=<some OpenType math font> \textfont2=<some OpenType math font> \textfont3=<some OpenType math font> ... to produce a minimal broken example.

The problem

latex2e/required/amsmath/amsmath.dtx

Line 732 in e09c9d4

#3\fontdimen6#3\tw@

When \Umathfractiondelsize\displaystyle=\z@, the above expands to 2.39\fontdimen62.39\tw@, which is ridiculous.

The fix

% LuaTeX version
% \changes{...}{2020/11/..}{correct fontdimen syntax}
% |#1| is the delimiter, |#2| is the math style, |#3| is the math font,
% and |#4| is the fontdimen parameter.
%    \begin{macrocode}
\def\genfrac@rule#1#2#3#4{%
\hbox{$\left#1\vcenter{\hrule \@width\z@
                       \@height
                       \ifdim\Umathfractiondelsize#2=\z@
                       #4\fontdimen6#3\tw@
                       \else
                       \Umathfractiondelsize#2%
                       \fi
                      }\right.$}}
%    \end{macrocode}
%
%    \begin{macrocode}
\def\genfrac@choice#1#2{%
\ifx @#2@\else
%    \end{macrocode}
% \changes{v2.17a}{2017/09/02}{move \cs{nulldelimiterspace} correction}
%    \begin{macrocode}
\ifx c#1\kern-\nulldelimiterspace\fi
{\delimitershortfall\z@\delimiterfactor\@m
 \mathsurround\z@\nulldelimiterspace\z@
%    \end{macrocode}
% \changes{...}{2020/11/..}{correct fontdimen syntax}
%    \begin{macrocode}
\mathchoice
{\genfrac@rule{#2}\displaystyle\textfont{2.39}}%
{\genfrac@rule{#2}\textstyle\textfont{1}}%
{\genfrac@rule{#2}\scriptstyle\scriptfont{1.45}}%
{\genfrac@rule{#2}\scriptscriptstyle\scriptscriptfont{1.35}}%
}%
\ifx o#1\kern-\nulldelimiterspace\fi
\fi
}
%    \end{macrocode}

[davidcarlisle]

Yes thanks, not sure how that slipped through.

[RuixiZhang42]

@davidcarlisle I had a theory why it slipped through: \Umathfractiondelsize<mathstyle>=\z@ is so unlikely that no one actually encounter this situation. In fact, if we have (a) LuaTeX+TFM fonts, then \Umathfractiondelsize<mathstyle> comes from TFM parameters (and are non-zero); if we have (b) LuaTeX+OT fonts, then the unicode-math package (or maybe luaotfload?) patches \fontdimen20 and \fontdimen21 automatically (again, providing non-zero values). The only way to have \Umathfractiondelsize<mathstyle>=\z@ is either setting OT math fonts without unicode-math, or deliberately declaring \Umathfractiondelsize<mathstyle>=0pt somewhere in the document. Either approach seems unlikely to come from an average LaTeX user.

[RuixiZhang42]

@davidcarlisle Another thing while I was digging: Why is the \textstyle parameter {1} instead of {1.01}?

According to cmsy10.mf, the \fontdimen21 is deliberately 10.1pt#, so that in textstyle a binomial coefficient will be forced to use \big-version of parentheses (which are 12pt tall). If \fontdimen21 were just 10pt#, the parentheses would be too small to cover a stacked \atop fraction (7pt numerator + 7pt denominator).

But amsmath codes \genfrac@rule{#2}{21}\textfont{1} (XeTeX) and \genfrac@rule{#2}\textstyle\textfont{1} (fixed LuaTeX). IMO they should be

\genfrac@rule{#2}{21}\textfont{1.01}       % for XeTeX
\genfrac@rule{#2}\textstyle\textfont{1.01} % for LuaTeX

[RuixiZhang42]

A sample test file:

\documentclass{article}
\usepackage{amsmath}

%     One em quad in cmsy10 is 1.000003 of design size,
% i.e., 10pt cmsy10 has \fontdimen6=655361sp=10.00002pt.
% This hides the inappropriate
% \genfrac@rule{#2}{21}\textfont{1}       for XeTeX, and
% \genfrac@rule{#2}\textstyle\textfont{1} for LuaTeX.
%     To highlight these two hidden bugs, we load cmsy10
% slightly scaled down for this particular test.
\DeclareFontShape{OMS}{cmsy}{m}{n}{%
      <5><6><7><8><9>gen*cmsy%
      <10>*[0.99999]cmsy10% load cmsy10 slightly scaled down
      <10.95><12><14.4><17.28><20.74><24.88>cmsy10%
      }{}

\begin{document}

$\displaystyle     \binom{n}{k},
 \textstyle        \binom{n}{k},
 \scriptstyle      \binom{n}{k},
 \scriptscriptstyle\binom{n}{k}$

\ifx\directlua\undefined
  \ifx\XeTeXcharclass\undefined
  \else
    \fontdimen20\textfont2=0pt         \fontdimen21\textfont2=0pt %
    \fontdimen20\scriptfont2=0pt       \fontdimen21\scriptfont2=0pt %
    \fontdimen20\scriptscriptfont2=0pt \fontdimen21\scriptscriptfont2=0pt %
  \fi
\else
    \Umathfractiondelsize\displaystyle=0pt %
    \Umathfractiondelsize\textstyle=0pt %
    \Umathfractiondelsize\scriptstyle=0pt %
    \Umathfractiondelsize\scriptscriptstyle=0pt %
\fi

$\displaystyle     \binom{n}{k},
 \textstyle        \binom{n}{k},
 \scriptstyle      \binom{n}{k},
 \scriptscriptstyle\binom{n}{k}$

\end{document}

[RuixiZhang42]

Hmm…

---

[RuixiZhang42]

@davidcarlisle Two additional fixes:

1. The \hbox{$\left#1\vcenter{... should have a mathstyle preceding \left#1, otherwise the axis height is wrong.
2. I changed 2.39, 1, 1.45, 1.35 to 2.39, 1.01, 0.988, 0.964 to match cmsy10, cmsy7 and cmsy5.

Now the test file below produces identical(*) results across 3 different engines.
*Well, XeTeX agrees with pdfTeX perfectly (which is nice), but LuaTeX gives too small parens for \scriptstyle and \scriptscriptstyle. (I think this is an engine bug. Notice how the LuaTeX script versions have thinner parens—they look like coming from different fonts)

\documentclass{article}
\usepackage{amsmath}

%     One em quad in cmsy10 is 1.000003 of design size,
% i.e., 10pt cmsy10 has \fontdimen6=655361sp=10.00002pt.
% This hides the inappropriate
% \genfrac@rule{#2}{21}\textfont{1}       for XeTeX, and
% \genfrac@rule{#2}\textstyle\textfont{1} for LuaTeX.
%     To highlight these two hidden bugs, we load cmsy10
% slightly scaled down for this particular test.
\DeclareFontShape{OMS}{cmsy}{m}{n}{%
      <5><6><7><8><9>gen*cmsy%
      <10>*[0.99999]cmsy10% load cmsy10 slightly scaled down
      <10.95><12><14.4><17.28><20.74><24.88>cmsy10%
      }{}

\makeatletter
\ifx\directlua\@undefined
\ifx\XeTeXcharclass\@undefined
\else
% XeTeX version
% \cs{genfrac@rule} takes 5 arguments: \arg{1} delimiter,
% \arg{2} mathstyle, \arg{3} fontdimen index, \arg{4} font,
% \arg{5} fontdimen value (taken to match cmsy10, cmsy7, cmsy5).
\def\genfrac@rule#1#2#3#4#5{%
\hbox{$#2% <--- mathstyle here to get the correct axis height
       \left#1\vcenter{\hrule \@width\z@
                       \@height
                       \ifdim\fontdimen#3#4\tw@=\z@
                       #5\fontdimen6#4\tw@
                       \else
                       \fontdimen#3#4\tw@
                       \fi
                      }\right.$}}
\def\genfrac@choice#1#2{%
\ifx @#2@\else
\ifx c#1\kern-\nulldelimiterspace\fi
{\delimitershortfall\z@\delimiterfactor\@m
 \mathsurround\z@\nulldelimiterspace\z@
\mathchoice
{\genfrac@rule{#2}\displaystyle{20}\textfont{2.39}}%
{\genfrac@rule{#2}\textstyle{21}\textfont{1.01}}%
{\genfrac@rule{#2}\scriptstyle{21}\scriptfont{0.988}}%
{\genfrac@rule{#2}\scriptscriptstyle{21}\scriptscriptfont{0.964}}%
}%
\ifx o#1\kern-\nulldelimiterspace\fi
\fi
}
\fi
\else
% LuaTeX version
\def\genfrac@rule#1#2#3#4{%
\hbox{$#2% <--- mathstyle here to get the correct axis height
       \left#1\vcenter{\hrule \@width\z@
                       \@height
                       \ifdim\Umathfractiondelsize#2=\z@
                       #4\fontdimen6#3\tw@
                       \else
                       \Umathfractiondelsize#2%
                       \fi
                      }\right.$}}
\def\genfrac@choice#1#2{%
\ifx @#2@\else
\ifx c#1\kern-\nulldelimiterspace\fi
{\delimitershortfall\z@\delimiterfactor\@m
 \mathsurround\z@\nulldelimiterspace\z@
\mathchoice
{\genfrac@rule{#2}\displaystyle\textfont{2.39}}%
{\genfrac@rule{#2}\textstyle\textfont{1.01}}%
{\genfrac@rule{#2}\scriptstyle\scriptfont{0.988}}%
{\genfrac@rule{#2}\scriptscriptstyle\scriptscriptfont{0.964}}%
}%
\ifx o#1\kern-\nulldelimiterspace\fi
\fi
}
\fi
\makeatother

\begin{document}

$\displaystyle     \binom{n}{k},
 \textstyle        \binom{n}{k},
 \scriptstyle      \binom{n}{k},
 \scriptscriptstyle\binom{n}{k}$

\ifx\directlua\undefined
  \ifx\XeTeXcharclass\undefined
  \else
    \fontdimen20\textfont2=0pt         \fontdimen21\textfont2=0pt %
    \fontdimen20\scriptfont2=0pt       \fontdimen21\scriptfont2=0pt %
    \fontdimen20\scriptscriptfont2=0pt \fontdimen21\scriptscriptfont2=0pt %
  \fi
\else
    \Umathfractiondelsize\displaystyle=0pt %
    \Umathfractiondelsize\textstyle=0pt %
    \Umathfractiondelsize\scriptstyle=0pt %
    \Umathfractiondelsize\scriptscriptstyle=0pt %
\fi

$\displaystyle     \binom{n}{k},
 \textstyle        \binom{n}{k},
 \scriptstyle      \binom{n}{k},
 \scriptscriptstyle\binom{n}{k}$

\end{document}

[FrankMittelbach]

@davidcarlisle tell me not if that assignment is premature :-)

[zauguin]

@RuixiZhang42

Well, XeTeX agrees with pdfTeX perfectly (which is nice), but LuaTeX gives too small parens for \scriptstyle and \scriptscriptstyle. (I think this is an engine bug. Notice how the LuaTeX script versions have thinner parens—they look like coming from different fonts)

They come from different font, but the change in LuaTeX seems intentional. What happens is the following: In TeX, delimiters come in a small and a big variant. If the small variant (and all successor glyphs) is too small, TeX tries the small variant of bigger mathstyles. If all of them are too small, it finally tries the big version (and it's successors). (Again, this falls back to bigger mathstyles if it is too small, but since the last successor if often extensible this rarely happens)

LuaTeX on the other hand never looks at other mathstyles. So effectivly, the parens in other engines are the small \textstyle parens, while the LuaTeX uses big \scriptstyle/\scriptscrptstyle parens.

While this change does not seem to be documented, it is consistent with LuaTeX's general tendency not to use the small/big delimiter distinction so much.

[stale]

This issue has been automatically marked as stale because it has not had recent activity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity.

[stale]

This issue has been automatically marked as stale because it has not had recent activity.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.