fix #17260 render \ properly in nim doc, rst2html

[timotheecour]

/cc @a-mr

• fix nim doc/rst2html renders \v, \\, etc incorrectly #17260
• unblocks RST backtick refactor manual.rst #17259 and RST backtick refactor (all *.rst except manual.rst and rst_examples.rst) #17258 (/cc @quantimnot)

before PR:

https://nim-lang.github.io/Nim/system.html#addEscapedChar%2Cstring%2Cchar

after PR:

note 1

the fix for this should not be controversial:

`/x` (`/` not followed by a backtick)
`//` (ditto, the `/` is paired)

note 2

the fix for this is more controversial but IMO is the correct approach:
I'm treating this as legal:

`/` (`/` followed by a backtick)

because:

• it's legal in markdown and is what users will expect (and in fact nim docs already uses it even though, prior to this PR, it didn't render properly, eg see addEscapedChar docs)
• nim rst spec technically allows it, although it's vague about it see https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#interpreted-text

Interpreted text is text that is meant to be related, indexed, linked, summarized, or otherwise processed, but the text itself is typically left alone. Interpreted text is enclosed by single backquote characters:

• as mentioned in nim doc/rst2html renders \v, \\, etc incorrectly #17260 (comment):

The RST spec does not cover escaping in interpreted text explicitly but the idea is mostly clear.
The code role is not really documented about that behavior

• the rst behavior has design flaws / limitations, eg code directive with single backslash character not interpreted as expected sphinx-doc/sphinx#8396

• this does behave differently from rst2html as noted there

• backticks are rarely needed, eg we can use:

## see `system.[]=` # no need to render a backtick inside the backtick pair

for simpler doc links (nim-lang/RFCs#125) this also shouldn't be a problem:

## see `system.[]=`_

note 3

for the rare cases where we'll want to show a backtick character in docs (outside of code blocks/runnableExamples where this works fine), eg to distinguish case from `case` , we should use markdown syntax:

foo1 ``` ` ``` # strips leading and trailing space and we end up with a single backtick rendered
foo2 ``` `` ``` # ditto, we end up with 2 backticks
foo2 ```` ``` ```` # ditto, we end up with 3 backticks

it's flexible and has saner escaping semantics (just increase the number of surrounding backticks)

future work

• (refs RST minor bugs #17340 (comment)) as mentioned in https://stackoverflow.com/a/66557930/1426932 \| inside a table cell should render as |, not \| (but outside a table cell, it should render as in this PR); I've edited the test to make it pass but in future work we should address this edge case

| A1 header    | A2 \| not fooled
| :---         | ----:       |
| C1           | C2 **bold** | ignored |
| D1 `code \|` | D2          | also ignored
| E1 \| text   |
|              | F2 without pipe
not in table

there are workarounds until this is addressed, eg:
split the rendered pipe in 2:

`x` \| `y`

or using a different unicode char for the pipe https://stackoverflow.com/a/43070960/1426932

[a-mr]

I don't understand your vigor for exact replicating Markdown behavior.

Using code as the default role is fine. (the fact that python rst2html has title-reference, which is italic text, as the default role makes little sense to me.) But it means we should be able input backticks there with \`.

backticks are rarely needed, eg we can use:

The cases when we need a standalone \ are much more rare. We can just use double ticks for that rare case:

``\``

is what users will expect (and in fact nim docs already uses it even though, prior to this PR, it didn't render properly, eg see addEscapedChar docs)

You are a little cunning here. This bug that you showed was actually introduced in double backtick elimination in d447c0f , previously it was rendered OK with double backticks.

unblocks #17259 and #17258

For the time being you can just use double backticks wherever a backslash is present.

[Araq]

after PR:

What? It's much worse after your PR. Why would there be double backslashes? o.O

[a-mr]

@Araq
it's just that @timotheecour stumbled upon a bad example accidentally. The original text was written in this confusing way in 8caf257 indeed.

The essence of this PR is that after the recent double-backtick elimination any changed markup containing backslashes got broken because backslashes are interpreted differently in single backticks. So @timotheecour tries to hotfix that. Which is also dangerous because there can exist markup in single backticks that relies on old behavior.

[Araq]

So undo the commits that use single backticks everywhere.

[konsumlamm]

So undo the commits that use single backticks everywhere.

Why not just undo the changes where it breaks something? That should only be a few changes (most of the time, backticks are used around variable names). IMO it's not worth it to sacrifice using single backticks.

[timotheecour]

What? It's much worse after your PR. Why would there be double backslashes? o.O

@Araq I don't understand this comment. The double backslashes in the rendered html are correct.
In fact it matches 1.4 docs: https://nim-lang.org/docs/system.html#addEscapedChar%2Cstring%2Cchar

There is a double backslash because addEscapedChar adds a backslash to something like \n, otherwise it'd be indistinguishable from \n.

replaces \n by \\n # correct: after this PR, as well as in 1.4 docs
replaces n by \n # incorrect: in devel docs
replaces \n by \n # incorrect (following your suggestion of no double backslashes) => nothing replaced

whether this should written in the doc comment as

## replaces `\n` by `\\n` (as enabled by this PR)
or
## replaces ``\n`` by ``\\n`` (as was needed in 1.4 to get the desired html, but still possible after this PR)

is a different question, but your suggestion that there should be no double backslashes here is incorrect.

The original text was written in this confusing way in 8caf257 indeed.

I don't see how the original text (in the rendered html) from 8caf257 was confusing, it conveys exactly what the proc does. This is consistent with other uses in the manual and elsewhere in that \n means newline, and \\n means backslash followed by n (total 2 chars)

[Araq]

@Araq I don't understand this comment. The double backslashes in the rendered html are correct.

Ok, ok, I take it back. That doesn't mean that I can review this, I cannot, I have no idea what changed and why.

[narimiran]

The example from the addEscapedChar documentation should also be added to nimdoc/rst1html/source/rst_examples.rst — once the escaping is fixed, we don't want to have those regressions again.

[timotheecour]

PTAL, I'm now honoring these:

``foo`` # double backtick behavior unchanged
`fo\a\\b\`o` # single backtick now renders as: fo\a\\b`o (the backtick is escaped, but not the other backslashes)

The example from the addEscapedChar documentation should also be added to

I've added tests to tests/stdlib/trstgen.nim; before this PR, there were no tests for interpreted text or inline literals

[Araq]

@narimiran's turn.

[timotheecour]

ping @narimiran