rewrite the regex-syntax crate #452
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit represents a ground up rewrite of the regex-syntax crate.
This commit is also an intermediate state. That is, it adds a new
regex-syntax-2 crate without making any serious changes to any other
code. Subsequent commits will cover the integration of the rewrite and
the removal of the old crate.
The rewrite is intended to be the first phase in an effort to overhaul
the entire regex crate. To that end, this rewrite takes steps in that
direction:
* The principle change in the public API is an explicit split between a
regular expression's abstract syntax (AST) and a high-level
intermediate representation (HIR) that is easier to analyze. The old
version of this crate mixes these two concepts, but leaned heavily
towards an HIR. The AST in the rewrite has a much closer
correspondence with the concrete syntax than the old `Expr` type does.
The new HIR embraces its role; all flags are now compiled away
(including the `i` flag), which will simplify subsequent passes,
including literal detection and the compiler. ASTs are produced by
ast::parse and HIR is produced by hir::translate. A top-level parser
is provided that combines these so that callers can skip straight from
concrete syntax to HIR.
* Error messages are vastly improved thanks to the span information that
is now embedded in the AST. In addition to better formatting, error
messages now also include helpful hints when trying to use features
that aren't supported (like backreferences and look-around). In
particular, octal support is now an opt-in option. (Octal support
will continue to be enabled in regex proper to support backwards
compatibility, but will be disabled in 1.0.)
* More robust support for Unicode Level 1 as described in UTS#18.
In particular, we now fully support Unicode character classes
including set notation (difference, intersection, symmetric
difference) and correct support for named general categories, scripts,
script extensions and age. That is, `\p{scx:Hira}` and `p{age:3.0}`
now work. To make this work, we introduce an internal interval set
data structure.
* With the exception of literal extraction (which will be overhauled in
a later phase), all code in the rewrite uses constant stack space,
even while performing analysis that requires structural induction over
the AST or HIR. This is done by pushing the call stack onto the heap,
and is abstracted by the `ast::Visitor` and `hir::Visitor` traits.
The point of this method is to eliminate stack overflows in the
general case.
* Empty sub-expressions are now properly supported. Expressions like
`()`, `|`, `a|` and `b|()+` are now valid syntax.
The principle downsides of these changes are parse time and binary size.
Both seemed to have increased (slower and bigger) by about 1.5x. Parse
time is generally peanuts compared to the compiler, so we mostly don't
care about that. Binary size is mildly unfortunate, and if it becomes a
serious issue, it should be possible to introduce a feature that
disables some level of Unicode support and/or work on compressing the
Unicode tables. Compile times have increased slightly, but are still a
very small fraction of the overall time it takes to compile `regex`.
Fixes rust-lang#174, Fixes rust-lang#424
|
I'm a fan of the Visitor! |
This commit moves the entire regex crate over to the regex-syntax-2 rewrite. Most of this is just rewriting types. The compiler got the most interesting set of changes. It got simpler in some respects, but not significantly so.
With the regex syntax rewrite, we now support empty subexpressions more officially. Unfortunately, the compiler has trouble with empty subexpressions in alternation branches. There's no particular reason to not support for them, but they are difficult/awkward to express with the current compiler. So just ban them for now. If one does need an empty subexpression in an alternate branch, then amusingly, something like `()?|z` will work. We could rewrite all such empty sub-expressions into `()?`, which would retain the same match semantics, but we choose to take the most conservative change possible.
This commit provides exhaustive documentation for the regex crate's support
for Level 1 ("Basic Unicode Support") as documented in UTS#18.
We also document the small number of additions added to the concrete
syntax as a result of the regex-syntax rewrite.
See: http://unicode.org/reports/tr18/
This sub-command prints out the UTF-8 alternation machine for an arbitrary character class.
This commit adds an explicit Debug impl for regex's main Error type. The purpose of this impl is to format parse errors in normal panic messages more nicely. This is slightly idiosyncratic, but the default Debug impl prints the full string anyway, we might as well format it nicely. See also: rust-lang#450
This commit does the mechanical changes necessary to remove the old regex-syntax crate and replace it with the rewrite. The rewrite now subsumes the `regex-syntax` crate name, and gets a semver bump to 0.5.0.
BurntSushi
force-pushed
the
syntax-rewrite
branch
from
34d591b to
a75ed45
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR represents a ground up rewrite of the regex-syntax crate. The
rewrite is intended to be the first phase in an effort to overhaul
the entire regex crate. To that end, this rewrite takes steps in that
direction:
between a regular expression's abstract syntax (AST) and a high-level
intermediate representation (HIR) that is easier to analyze. The
old version of this crate mixes these two concepts, but leaned
heavily towards an HIR. The AST in the rewrite has a much closer
correspondence with the concrete syntax than the old
Exprtypedoes. The new HIR embraces its role; all flags are now compiled away
(including the
iflag), which will simplify subsequent passes,including literal detection and the compiler. ASTs are produced by
ast::parse and HIR is produced by hir::translate. A top-level parser
is provided that combines these so that callers can skip straight
from concrete syntax to HIR.
is now embedded in the AST. In addition to better formatting, error
messages now also include helpful hints when trying to use features
that aren't supported (like backreferences and look-around). In
particular, octal support is now an opt-in option. (Octal support
will continue to be enabled in regex proper to support backwards
compatibility, but will be disabled in 1.0.)
is now documented exhaustively in a new
UNICODE.mddocument. Inparticular, we now fully support Unicode character classes including
set notation (difference, intersection, symmetric difference) and
correct support for named general categories, scripts, script
extensions and age. That is,
\p{scx:Hira}andp{age:3.0}nowwork. To make this work, we introduce an internal interval set data
structure.
a later phase), all code in the rewrite uses constant stack space,
even while performing analysis that requires structural induction over
the AST or HIR. This is done by pushing the call stack onto the heap,
and is abstracted by the
ast::Visitorandhir::Visitortraits.The point of this method is to eliminate stack overflows in the
general case.
(),|,a|andb|()+are now valid syntax. (Note that we banempty sub-expressions in alternates in the regex compiler for now.)
The principle downsides of these changes are parse time and binary
size. Both seemed to have increased (slower and bigger) by about
1.5x-2x. Parse time is generally peanuts compared to the compiler, so
we mostly don't care about that. Binary size is mildly unfortunate,
and if it becomes a serious issue, it should be possible to introduce
a feature that disables some level of Unicode support and/or work on
compressing the Unicode tables. Compile times have increased slightly,
but are still a very small fraction of the overall time it takes to
compile
regex.Fixes #174, Fixes #424