Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-125008: Fix tokenize.untokenize roundtrip for \n{{ #125013

Merged
merged 4 commits into from
Oct 6, 2024

Conversation

tomasr8
Copy link
Member

@tomasr8 tomasr8 commented Oct 5, 2024

I'm not sure if this is the best way to fix this issue so I'm open to alternatives :)

@wyattscarpenter
Copy link

Might as well add test cases for \ alone, while you're at it. Hope this is helpful, if you do decide to do that:

        self.check_roundtrip(r"rf'\t{{foo}}'")
        self.check_roundtrip(r"f'\\t{{foo}}'")
        self.check_roundtrip(r"rf'\\\t{{foo}}'")
        self.check_roundtrip(r"f'\\\\t{{foo}}'")

(some of them are raw to avoid SyntaxWarning: invalid escape sequence '\{'.

@pablogsal
Copy link
Member

The changes LGTM. Thanks a lot for the catch and the fix @tomasr8!

I will wait if you want to address @wyattscarpenter suggestion and we can land afterwards

@pablogsal pablogsal added awaiting review needs backport to 3.12 bug and security fixes needs backport to 3.13 bugs and security fixes and removed awaiting merge labels Oct 6, 2024
@pablogsal
Copy link
Member

This will need backport to 3.12 and 3.13

@tomasr8
Copy link
Member Author

tomasr8 commented Oct 6, 2024

Thanks for the suggestion @wyattscarpenter ! I added more tests with raw strings :)

@wyattscarpenter
Copy link

wyattscarpenter commented Oct 6, 2024

Sorry, I bungled the code example I was trying to give you. I actually meant

        self.check_roundtrip(r"rf'\{{foo}}'")
        self.check_roundtrip(r"f'\\{{foo}}'")
        self.check_roundtrip(r"rf'\\\{{foo}}'")
        self.check_roundtrip(r"f'\\\\{{foo}}'")

\\ currently suffers from the same defect of the disappearing {, like \t, but it's sort of a special case so it might be worth testing for explicitly. Or not; it's not that special of a case so maybe it doesn't matter.

@tomasr8
Copy link
Member Author

tomasr8 commented Oct 6, 2024

Added more tests :) f-strings are tricky so I think it's better to have more tests anyway

@pablogsal pablogsal merged commit db23b8b into python:main Oct 6, 2024
34 checks passed
@miss-islington-app
Copy link

Thanks @tomasr8 for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 6, 2024
@pablogsal
Copy link
Member

Landing! Great job

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 6, 2024
@bedevere-app
Copy link

bedevere-app bot commented Oct 6, 2024

GH-125020 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Oct 6, 2024
@bedevere-app
Copy link

bedevere-app bot commented Oct 6, 2024

GH-125021 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.12 bug and security fixes label Oct 6, 2024
@tomasr8 tomasr8 deleted the untokenize branch October 6, 2024 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants