Tokenize does not roundtrip {{ after \n

[wyattscarpenter]

Bug report

Bug description:

import tokenize, io
source_code = r'''
f"""{80 * '*'}\n{{test}}{{details}}{{test2}}\n{80 * '*'}"""
'''
tokens = tokenize.generate_tokens(io.StringIO(source_code).readline)
x = tokenize.untokenize((t,s) for t, s, *_ in tokens)
print(x)

Expected:

f"""{80 *'*'}\n{{test}}{{details}}{{test2}}\n{80 *'*'}"""

Got:

f"""{80 *'*'}\n{test}}{{details}}{{test2}}\n{80 *'*'}"""

Note the absence of a second { in the {{ after the \n — but in no other positions.

Unlike some other roundtrip failures of tokenize, some of which are minor infelicities, this one actually creates a syntactically invalid program on roundtrip, which is quite bad. You get a SyntaxError: f-string: single '}' is not allowed when trying to use the results.

CPython versions tested on:

3.12

Operating systems tested on:

Linux, Windows

Linked PRs

• gh-125008: Fix tokenize.untokenize roundtrip for \n{{ #125013
• [3.13] gh-125008: Fix tokenize.untokenize roundtrip for \n{{ (GH-125013) #125020
• [3.12] gh-125008: Fix tokenize.untokenize roundtrip for \n{{ (GH-125013) #125021

[wyattscarpenter]

Furthermore, here is the output of the following code:

import tokenize, io
source_code = r'f"\n{{test}}"'
tokens = tokenize.generate_tokens(io.StringIO(source_code).readline)
for t in tokens:
  print(t)

TokenInfo(type=61 (FSTRING_START), string='f"', start=(1, 0), end=(1, 2), line='f"\\n{{test}}"')
TokenInfo(type=62 (FSTRING_MIDDLE), string='\\n{', start=(1, 2), end=(1, 5), line='f"\\n{{test}}"')
TokenInfo(type=62 (FSTRING_MIDDLE), string='test}', start=(1, 6), end=(1, 11), line='f"\\n{{test}}"')
TokenInfo(type=63 (FSTRING_END), string='"', start=(1, 12), end=(1, 13), line='f"\\n{{test}}"')
TokenInfo(type=4 (NEWLINE), string='', start=(1, 13), end=(1, 14), line='f"\\n{{test}}"')
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')

So, it seems that the line is getting in alright, but the \n{{ is getting turned into a \n{ in the tokenizer somehow.

Same erroneous output for the bytes version (with rb-string, BytesIO and tokenize.tokenize).

[AlexWaygood]

It looks like this was a regression in Python 3.12; I can't reproduce the behaviour with Python 3.11. I'm guessing it was caused by the PEP-701 changes.

[AlexWaygood]

Reproduced on the main branch as well.

[tomasr8]

This seems to happen with other escape characters as well:

import tokenize, io
source_code = r'f"""\t{{test}}"""'
tokens = tokenize.generate_tokens(io.StringIO(source_code).readline)
x = tokenize.untokenize((t,s) for t, s, *_ in tokens)
print(x)  # f"""\t{test}}"""

import tokenize, io
source_code = r'f"""\r{{test}}"""'
tokens = tokenize.generate_tokens(io.StringIO(source_code).readline)
x = tokenize.untokenize((t,s) for t, s, *_ in tokens)
print(x)  # f"""\r{test}}"""

[tomasr8]

I think the issue is in this method:

cpython/Lib/tokenize.py

Lines 187 to 208 in 16cd6cc

	def escape_brackets(self, token):
	characters = []
	consume_until_next_bracket = False
	for character in token:
	if character == "}":
	if consume_until_next_bracket:
	consume_until_next_bracket = False
	else:
	characters.append(character)
	if character == "{":
	n_backslashes = sum(
	1 for char in _itertools.takewhile(
	"\\".__eq__,
	characters[-2::-1]
	)
	)
	if n_backslashes % 2 == 0:
	characters.append(character)
	else:
	consume_until_next_bracket = True
	characters.append(character)
	return "".join(characters)

This PR fixed the handling of Unicode literals (e.g. \\N{foo}), but it seems to only
be checking for the presence of backslashes without checking if they are followed by N. This appears to fix that:

            if character == "{":
                n_backslashes = sum(
                    1 for char in _itertools.takewhile(
                        "\\".__eq__,
                        characters[-2::-1]
                    )
                )
-               if n_backslashes % 2 == 0:
+               if n_backslashes % 2 == 0 or characters[-1] != "N":
                    characters.append(character)
                else:
                    consume_until_next_bracket = True

[wyattscarpenter]

Extremely late for me to say this, but I thought I'd add: I only unearthed this bug because I'm working on code that incidentally tries to tokenizer-roundtrip everything in https://github.com/hauntsaninja/mypy_primer. So, doing something like that (possibly literally just that) as a test case for tokenize could perhaps be a good idea to prevent future regressions — although setting that up sounds like a hassle!

[tomasr8]

We actually already do that here for some random files in the test folder:

cpython/Lib/test/test_tokenize.py

Lines 1822 to 1835 in 26d6277

	class TestRoundtrip(TestCase):
	def check_roundtrip(self, f):
	"""
	Test roundtrip for `untokenize`. `f` is an open file or a string.
	The source code in f is tokenized to both 5- and 2-tuples.
	Both sequences are converted back to source code via
	tokenize.untokenize(), and the latter tokenized again to 2-tuples.
	The test fails if the 3 pair tokenizations do not match.
	When untokenize bugs are fixed, untokenize with 5-tuples should
	reproduce code that does not contain a backslash continuation
	following spaces. A proper test should test this.
	"""

Though, so far it only compares the tokens, not the actual source code. I'd like to extend this check to compare the source code as well here: #126010