Catcode of ampersand & when mathescape is used

[muzimuzhi]

Currenly mathescape restores catcode for $, ^, and _, but math material may also contain & in an math array, see https://tex.stackexchange.com/q/625732.

Note the mathescape option from listings package already restores catcode of & between $...$.

\documentclass{article}
\usepackage{amsmath}
\usepackage{listings}
\usepackage{minted}

\begin{document}

\subsection*{Actual}

\begin{Verbatim}[commandchars=\\\{\}, mathescape]
# catcode of & = $\text{\the\catcode`\&}, \begin{vmatrix} a & b \\ c & d \end{vmatrix}$
\end{Verbatim}

\makeatletter
% This patches fvextra option "mathescape", but unfortunately it won't work
% for minted, since the pygments-generated (till v2.16.1) "Verbatim" env
% always drop "mathescape" option and use its own "codes={...}" settings.
%
% See https://github.com/pygments/pygments/issues/1988
\appto\FV@MathEscape{\catcode`\&=4\relax}
\makeatother

\subsection*{With workaround}

% NOTE:
% Using "mathescape" now causes all occurances of $, ^, _, and in addition &
% gain their normal catcodes in verbatim content.
\begin{Verbatim}[commandchars=\\\{\}, mathescape, breaklines]
# Verbatim with mathescape, now \& outside of \$...\$ has to be escaped
# catcode of \& = $\text{\the\catcode`\&}, \begin{vmatrix} a & b \\ c & d \end{vmatrix}$
\end{Verbatim}

\begin{Verbatim}[breaklines]
# Verbatim, no mathescape
# catcode of & = $\text{\the\catcode`\&}, \begin{vmatrix} a & b \\ c & d \end{vmatrix}$
\end{Verbatim}

\subsection*{\Verb|lstlisting| env}
\begin{lstlisting}[basicstyle=\ttfamily, escapeinside=||, mathescape, breaklines]{python}
# catcode of & = $\text{\the\catcode`\&}, \begin{vmatrix} a & b \\ c & d \end{vmatrix}$
\end{lstlisting}

\end{document}

Proposal:

diff --git a/fvextra/fvextra.dtx b/fvextra/fvextra.dtx
index 612c885..c99e34b 100644
--- a/fvextra/fvextra.dtx
+++ b/fvextra/fvextra.dtx
@@ -4159,7 +4159,7 @@
 \define@booleankey{FV}{mathescape}%
  {\let\FancyVerbMathEscape\FV@MathEscape}%
  {\let\FancyVerbMathEscape\relax}
-\def\FV@MathEscape{\catcode`\$=3\catcode`\^=7\catcode`\_=8\relax}
+\def\FV@MathEscape{\catcode`\$=3\catcode`\&=4\catcode`\^=7\catcode`\_=8\relax}
 \FV@AddToHook\FV@CatCodesHook\FancyVerbMathEscape
 \fvset{mathescape=false}
 %    \end{macrocode}

[muzimuzhi]

Hmm the implementation of mathescape in fvextra is fragiler than that in listings. The one in fvextra restores char catcodes before the whole verbatim content is tokenized, instead of just in between $...$.

[gpoore]

Thanks for the suggestion. I've extended mathescape to add &. A new version of fvextra will be released very soon...I'm adding a lot of new features to support minted v3.0.

The current approach is more fragile than listings in the sense that the math-related catcodes are applied everywhere. However, if listings is only restoring catcodes between $...$, then its approach is probably fragile in the sense that it will fail when math is wrapped in a command (which forces tokenization before the opening $).

I suppose the best solution would be to tokenize with only the $ catcode restored, and then iterate through the tokens to find $...$. Then retokenize that range for &, ^, and _. And add in recursive retokenization within {...} when braces have normal group catcodes. But I'm not sure if mathescape gets used frequently enough to justify all that complexity.