support inlined nodes in grammar #500
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 25d9ede The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
OmarTawfik
force-pushed
the
add-inlined-productions
branch
from
3d9d62b to
b79f40c
Compare
Merged
AntonyBlakey
approved these changes
Jun 15, 2023
OmarTawfik
added a commit
that referenced
this pull request
Jun 16, 2023
- move parsing tokens to a common function - remove the need to pass token names as static strings everywhere, since we already have `strum::AsRefStr` deriver This decreases nesting, improves readability of the parser code, and decreases the generated code by roughly 30%. But most importantly, it unblocks #498 and #500 by making it easier to generate named or unnamed nodes at each parser root.
OmarTawfik
force-pushed
the
preserve-empty-nodes
branch
from
0f613cc to
7ff35c7
Compare
- Created `ProductionDefinition` enum to let parent `Production` hold common properties, similar to how `ParserDefinition`, `ScannerDefinition`, etc... work today. - Added `Production::inlined` boolean property, which defaults to `false` - Inlined productions no longer produce `ProductionKind`, `TokenKind`, or `RuleKind`. To test this end-to-end, I inlined a `Scanner`, a `Parser`, and a `PrecedenceParser` in the current grammar. Will follow up in the next few PRs with: - inlining additional nodes. - adding validation to make sure references between inlined nodes are valid, and that they are not leaked to public APIs.
OmarTawfik
force-pushed
the
add-inlined-productions
branch
from
b79f40c to
25d9ede
Compare
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jul 7, 2023
This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @nomicfoundation/[email protected] ### Minor Changes - [#502](#502) [`c383238`](c383238) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Added error recovery i.e. a CST is _always_ produced, even if there are errors. The erroneous/skipped text is in the CST as a `TokenKind::SKIPPED` token. - [#501](#501) [`cb221fe`](cb221fe) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - generate typescript string enums for CST kinds - [#517](#517) [`8bd5446`](8bd5446) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - extract inlined and sub-expressions in language grammar - [#518](#518) [`b3b562b`](b3b562b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fill in missing CST node names - [#515](#515) [`f24e873`](f24e873) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - switch over the NPM package to use CommonJS modules instead of ES modules. - [#498](#498) [`44f1ff7`](44f1ff7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - flatten unnamed CST nodes into parent nodes - [#502](#502) [`c383238`](c383238) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Use the Rowan model for the CST i.e. TokenNodes contain the string content, and RuleNodes contain only the combined _length_ of their children's text. - [#499](#499) [`1582d60`](1582d60) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - preserve correct ranges on empty rule nodes - [#500](#500) [`73ddac9`](73ddac9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - inlining CST nodes that offer no additional syntactic information - [#512](#512) [`72dc3d3`](72dc3d3) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Expression productions now correctly wrap the recursive 'calls' in a rule node ## [email protected] ### Minor Changes - [#502](#502) [`c383238`](c383238) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Added error recovery i.e. a CST is _always_ produced, even if there are errors. The erroneous/skipped text is in the CST as a `TokenKind::SKIPPED` token. - [#501](#501) [`cb221fe`](cb221fe) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - generate typescript string enums for CST kinds - [#517](#517) [`8bd5446`](8bd5446) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - extract inlined and sub-expressions in language grammar - [#518](#518) [`b3b562b`](b3b562b) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - fill in missing CST node names - [#515](#515) [`f24e873`](f24e873) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - switch over the NPM package to use CommonJS modules instead of ES modules. - [#498](#498) [`44f1ff7`](44f1ff7) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - flatten unnamed CST nodes into parent nodes - [#502](#502) [`c383238`](c383238) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Use the Rowan model for the CST i.e. TokenNodes contain the string content, and RuleNodes contain only the combined _length_ of their children's text. - [#499](#499) [`1582d60`](1582d60) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - preserve correct ranges on empty rule nodes - [#500](#500) [`73ddac9`](73ddac9) Thanks [@OmarTawfik](https://github.com/OmarTawfik)! - inlining CST nodes that offer no additional syntactic information - [#512](#512) [`72dc3d3`](72dc3d3) Thanks [@AntonyBlakey](https://github.com/AntonyBlakey)! - Expression productions now correctly wrap the recursive 'calls' in a rule node Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #323
ProductionDefinitionenum to let parentProductionhold common properties, similar to howParserDefinition,ScannerDefinition, etc... work today.Production::inlinedboolean property, which defaults tofalseProductionKind,TokenKind, orRuleKind.To test this end-to-end, I inlined a
Scanner, aParser, and aPrecedenceParserin the current grammar.Will follow up in the next few PRs with: