pgen instructions in readme

[fyquah]

I followed the instructions to get a working pgen binary and ran into the following compilation error:

Parser/grammar1.c: In function ‘PyGrammar_LabelRepr’:
Parser/grammar1.c:50:20: error: ‘_PyParser_TokenNames’ undeclared (first use in this function)
             return _PyParser_TokenNames[lb->lb_type];
                    ^
Parser/grammar1.c:50:20: note: each undeclared identifier is reported only once for each function it appears in
Parser/grammar1.c:61:1: warning: control reaches end of non-void function [-Wreturn-type]
 }

It is mentioned in the README that we need to overwrite Include/token.h, Include/compile.h, and Parser/tokenizer.c, I assume that's referring to replacing those in cpython with those in typed_ast/ast35 right?

[ddfisher]

Yes, that's correct. I think you may also need to unmangle the function names from Ta35* to Py*. It's pretty unfortunate -- I'm hoping to find a better way to get a working pgen at some point.

[ddfisher]

The files for pgen are now included, and pgen is compiled as part of ./tools/update_ast3_grammar