Crash when unpacking in yield

[drufat]

Use this sample file to reproduce the crash

def generator():
    a = ((1, 2), (3, 4))
    for i, tup in enumerate(a):
        yield i, *tup

print(list(generator()))

which should output

$ python test.py 
[(0, 1, 2), (1, 3, 4)]

Crash

$ yapf -i test.py 
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=16, value='*', context=(' ', (4, 17))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/yapf", line 11, in <module>
    load_entry_point('yapf==0.29.0', 'console_scripts', 'yapf')()
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 344, in run_main
    sys.exit(main(sys.argv))
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 216, in main
    changed = FormatFiles(
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 277, in FormatFiles
    changed |= _FormatFile(filename, lines, style_config, no_local_style,
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 298, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 85, in FormatFile
    reformatted_source, changed = FormatCode(
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.p#en2.parse.ParseError: bad input: type=16, value='*', context=(' ', (4, 17))

Versions

$ python --version
Python 3.8.2
$ yapf --version
yapf 0.29.0

[kamahen]

Which version of Python is used for yapf itself?

Crashes like this are usually caused by either syntactically incorrect input or by a recent change to the Python grammar that hasn't yet shown up in lib2to3 (which yapf uses). My guess it's the latter.
See this for more details: #772 (comment)

[kamahen]

The example code seems to use a Python 3.8 feature:

$ python3.8 /tmp/crash.py
[(0, 1, 2), (1, 3, 4)]

$ python3.7 /tmp/crash.py
  File "/tmp/crash.py", line 4
    yield i, *tup
             ^
SyntaxError: invalid syntax

$ cat /tmp/crash.py
def generator():
    a = ((1, 2), (3, 4))
    for i, tup in enumerate(a):
        yield i, *tup

print(list(generator()))

[drufat]

I only have python3.8 installed on my system. Both yapf and the example code run under that version.

[kamahen]

The problem isn't with Python 3.8, as such ... it's because Python 3.8's version of lib2to3, which yapf uses to parse your source, hasn't been updated with the latest grammar changes, and therefore can't parse "yield i, *tup" (it also can't parse "if a:=foo()". There isn't likely to be a solution until this bug is resolved: https://bugs.python.org/issue36541
A fix appears to exist, but it hasn't been incorporated into the Python release.
Once the fix is produced, it's possible that some additional changes will be needed to yapf, to handle the new grammar