bpo-40958: Avoid buffer overflow in the parser when indexing the current line

[pablogsal]

After thinking a bit more about the whole problem I found that we were processing the offsets incorrectly if the input is raw (no need to transform the offset to a character offset). Once we process the column offsets correctly, is quite straightforward to fix the incorrect access.

As an example of why we were processing the offset incorrectly previously, consider this code (current master without this PR):

❯ ./python
Python 3.9.0b1+ (heads/3.9:299d3d1c52, Jun  8 2020, 22:11:46)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
  File "<stdin>", line 1
    Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
                                  ^
SyntaxError: invalid syntax

while the old parser points correctly to the $ token:

❯ ./python -Xoldparser
Python 3.9.0b1+ (heads/3.9:299d3d1c52, Jun  8 2020, 22:11:46)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
  File "<stdin>", line 1
    Python = "Ṕýţĥòñ" + 34 $ lelummsmdsads +
                           ^
SyntaxError: invalid syntax

https://bugs.python.org/issue40958

[bedevere-bot]

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

[pablogsal]

I have made the requested changes; please review again

[bedevere-bot]

Thanks for making the requested changes!

@tiran: please review the changes made to this pull request.

[lysnikolaou]

Nice! 🎉

[lysnikolaou]

LGTM! Thanks thanks!

[pablogsal]

@tiran If you have some time today, could you confirm that this solves the ASAN/USAN problem for you? I have checked on my side that this is the case but it would be nice to have independent confirmation. I plan to land this today and do the backport to avoid potential merge conflicts.

[tiran]

@tiran If you have some time today, could you confirm that this solves the ASAN/USAN problem for you? I have checked on my side that this is the case but it would be nice to have independent confirmation. I plan to land this today and do the backport to avoid potential merge conflicts.

After a rebase to latest master the PR is passing asan/ubsan tests on my F32 machine.

[miss-islington]

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

[bedevere-bot]

GH-20919 is a backport of this pull request to the 3.9 branch.