[WIP] Detect several simple syntax errors in the parser #16308

ntBre · 2025-02-21T18:37:57Z

Summary

Currently stacked on #16090, this PR uses the framework there to detect several more simple version-related syntax errors in the parser:

Assignment expressions before 3.8
except* before 3.11
type statements before 3.12
type parameter lists before 3.12
type parameter defaults before 3.13

Test Plan

New CLI tests

I initially tried passing the target version to all of the parser functions, but this required a huge number of changes. The downside of the current approach is that we're likely to accumulate quite a large Vec<SyntaxError> in the parser, only to filter it down later. The upside is that we don't have to fix every single call site of every parser function right now

could be Copy as well, currently because Mode and PythonVersion are

This reverts commit 3f43c82. This didn't work because the fuzzing only runs with cfg(test) not cfg(fuzzing)

…rors-parser

github-actions · 2025-02-21T20:00:38Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

MichaReiser · 2025-02-22T08:05:17Z

This might have been your plan all along but I think I'd prefer a PR for each specific check vs one large PR that implements all of them. It should make them easier to review and I don't think it's necessary to ship them all at once

ntBre · 2025-02-22T18:37:04Z

Sounds good, I'll close this and split it up once #16257 and #16090 are merged since it's waiting on those anyway.

I also remove this in #16308, so even with additional variants we don't need the `match`. It was needed in an earlier draft of this prototype to emit a different syntax error as a ruff rule diagnostic

ntBre · 2025-02-24T21:34:31Z

I'll go ahead and close this. I absorbed some of the changes into #16090, and I'll add each of the new errors in its own PR.

This PR is the first in a series derived from #16308, each of which add support for detecting one version-related syntax error from #6591. This one should be the largest because it also includes a couple of additional changes: 1. the `syntax_errors!` macro, which makes adding more variants a bit easier 2. the `Parser::add_unsupported_syntax_error` method Otherwise I think the general structure will be the same for each syntax error: * Detecting the error in the parser * Inline parser tests for the new error * New ruff CLI tests for the new error Because of the second point here, this PR is currently stacked on #16357. As noted above, there are new inline parser tests, as well as new ruff CLI tests. Once #16379 is resolved, there should also be new mdtests for red-knot, but this PR does not currently include those.

## Summary This PR builds on the changes in #16220 to pass a target Python version to the parser. It also adds the `Parser::unsupported_syntax_errors` field, which collects version-related syntax errors while parsing. These syntax errors are then turned into `Message`s in ruff (in preview mode). This PR only detects one syntax error (`match` statement before Python 3.10), but it has been pretty quick to extend to several other simple errors (see #16308 for example). ## Test Plan The current tests are CLI tests in the linter crate, but these could be supplemented with inline parser tests after #16357. I also tested the display of these syntax errors in VS Code: ![image](https://github.com/user-attachments/assets/062b4441-740e-46c3-887c-a954049ef26e) ![image](https://github.com/user-attachments/assets/101f55b8-146c-4d59-b6b0-922f19bcd0fa) --------- Co-authored-by: Alex Waygood <[email protected]>

This PR is the first in a series derived from #16308, each of which add support for detecting one version-related syntax error from #6591. This one should be the largest because it also includes a couple of additional changes: 1. the `syntax_errors!` macro, which makes adding more variants a bit easier 2. the `Parser::add_unsupported_syntax_error` method Otherwise I think the general structure will be the same for each syntax error: * Detecting the error in the parser * Inline parser tests for the new error * New ruff CLI tests for the new error Because of the second point here, this PR is currently stacked on #16357. As noted above, there are new inline parser tests, as well as new ruff CLI tests. Once #16379 is resolved, there should also be new mdtests for red-knot, but this PR does not currently include those.

## Summary This PR is the first in a series derived from #16308, each of which add support for detecting one version-related syntax error from #6591. This one should be the largest because it also includes the addition of the `Parser::add_unsupported_syntax_error` method Otherwise I think the general structure will be the same for each syntax error: * Detecting the error in the parser * Inline parser tests for the new error * New ruff CLI tests for the new error ## Test Plan As noted above, there are new inline parser tests, as well as new ruff CLI tests. Once #16379 is resolved, there should also be new mdtests for red-knot, but this PR does not currently include those.

This PR is the first in a series derived from #16308, each of which add support for detecting one version-related syntax error from #6591. This one should be the largest because it also includes a couple of additional changes: 1. the `syntax_errors!` macro, which makes adding more variants a bit easier 2. the `Parser::add_unsupported_syntax_error` method Otherwise I think the general structure will be the same for each syntax error: * Detecting the error in the parser * Inline parser tests for the new error * New ruff CLI tests for the new error Because of the second point here, this PR is currently stacked on #16357. As noted above, there are new inline parser tests, as well as new ruff CLI tests. Once #16379 is resolved, there should also be new mdtests for red-knot, but this PR does not currently include those.

ntBre and others added 30 commits February 19, 2025 11:16

add unused Parser::syntax_errors field

6ba7166

add hard-coded python version and detect match

cab545f

make PythonVersion more public, convert SyntaxError in red-knot

8f0bd4c

process SyntaxErrors in ruff

846314f

pass tests

27e9461

add ruff test case

625682e

detect late future imports in the parser

e2614be

pass f404 tests

4423bd4

check if rules are enabled before converting to diagnostics

6ee2eb8

add a todo about duplicate diagnostics

a1b0aa6

clippy

5c24655

update to use ast::PythonVersion and Span

44a6c66

remove LateFutureImport detection from the parser

4bb914c

tidy up

1a1b2ad

clean up after rebase

723cf7e

add ParseOptions::target_version

832e0a7

filter syntax errors when parsing, store version on SyntaxError

d1daafa

make ParseOptions Clone

a0efadd

could be Copy as well, currently because Mode and PythonVersion are

pass ParseOptions::target_version in the linter

4dba857

pass ParseOptions::target_version in red-knot

a3f8ea9

ignore version-related syntax errors when fuzzing

3f43c82

Red-knot no longer panics!

3a5de09

Fix fuzzing script and fix Rust formatting

e47316e

Revert "ignore version-related syntax errors when fuzzing"

a50fa2a

This reverts commit 3f43c82. This didn't work because the fuzzing only runs with cfg(test) not cfg(fuzzing)

pass check_file_skips_type_checking_when by initializing settings

a156781

fix clippy lint about &impl ToString

114c207

Merge branch 'main' of github.com:astral-sh/ruff into brent/syntax-er…

5b55d04

…rors-parser

pass version to new parsed_module call

b06e1b1

WIP

2eecea8

ntBre added 3 commits February 21, 2025 14:33

add ExceptStarBeforePy311 variant, refactor SyntaxError::message

53ed4ac

use latest version ruff_linter_fixtures to avoid syntax errors

c832d1d

test except*

f5a607c

ntBre force-pushed the brent/simple-syntax-errors branch from ce9ab64 to f5a607c Compare February 21, 2025 19:51

ntBre added 4 commits February 21, 2025 15:20

factor out syntax_errors!, add add_syntax_error, and typebefore312

d6c2508

test type statement

8763bee

detect type parameter lists

832274b

detect type parameter defaults

35c7cd0

ntBre force-pushed the brent/syntax-errors-parser branch from b06e1b1 to cd90966 Compare February 24, 2025 14:55

ntBre mentioned this pull request Feb 24, 2025

Start detecting version-related syntax errors in the parser #16090

Merged

ntBre closed this Feb 24, 2025

ntBre mentioned this pull request Feb 25, 2025

Detect assignment expressions before Python 3.8 #16383

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Detect several simple syntax errors in the parser #16308

[WIP] Detect several simple syntax errors in the parser #16308

ntBre commented Feb 21, 2025 •

edited

Loading

github-actions bot commented Feb 21, 2025 •

edited

Loading

MichaReiser commented Feb 22, 2025 •

edited

Loading

ntBre commented Feb 22, 2025

ntBre commented Feb 24, 2025

[WIP] Detect several simple syntax errors in the parser #16308

[WIP] Detect several simple syntax errors in the parser #16308

Conversation

ntBre commented Feb 21, 2025 • edited Loading

Summary

Test Plan

github-actions bot commented Feb 21, 2025 • edited Loading

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

MichaReiser commented Feb 22, 2025 • edited Loading

ntBre commented Feb 22, 2025

ntBre commented Feb 24, 2025

ntBre commented Feb 21, 2025 •

edited

Loading

github-actions bot commented Feb 21, 2025 •

edited

Loading

`ruff-ecosystem` results

MichaReiser commented Feb 22, 2025 •

edited

Loading