Lex Jupyter Magic in assignment value position #30
Conversation
|
Current dependencies on/for this PR:
This comment was auto-generated by Graphite. |
parser/src/lexer.rs
Outdated
| @@ -1292,6 +1309,7 @@ where | |||
|
|
|||
| // Helper function to emit a lexed token to the queue of tokens. | |||
| fn emit(&mut self, spanned: Spanned) { | |||
| self.last_emitted = Some(spanned.0.clone()); | |||
There was a problem hiding this comment.
This seems a bit expensive as we'll be cloning every single token for the entire source code. Can we maybe look at the pending token instead?
There was a problem hiding this comment.
Yeah, that's very expensive because it even has to clone the heap allocations of the tokens.
Some optimisations: Only store the token kind (see ruff-python-ast TokenKind) which is only a u8.
Track the context explicitly:
https://github.com/mozilla/sweet.js/wiki/design and an implementation of it https://github.com/swc-project/swc/blob/026101b71e942ad2d7ff906cb68bf46345d77712/crates/swc_ecma_parser/src/lexer/state.rs#L756-L797
In your case it seems all you really care about is whether you've seen an equal token. But it would be nice if the chosen approach could eventually replace the SoftKeywordTokenizer as well.
There was a problem hiding this comment.
In your case it seems all you really care about is whether you've seen an
equaltoken.
Yes, I think having a boolean stating "Was the last token an Equal?" seems like the simplest option. I'll explore a bit around lexer context and state after this as it's pretty interesting :)
| pwd = !pwd | ||
| foo = %timeit a = b | ||
| bar = %timeit a % 3 | ||
| baz = %matplotlib \ | ||
| inline" |
There was a problem hiding this comment.
All of the valid positions for magic command token in terms of an assignment statement.
| let source = r" | ||
| # Other magic kinds are not valid here (can't test `foo = ?str` because '?' is not a valid token) | ||
| foo = /func | ||
| foo = ;func | ||
| foo = ,func | ||
|
|
||
| (foo == %timeit a = b) | ||
| (foo := %timeit a = b) | ||
| def f(arg=%timeit a = b): | ||
| pass" | ||
| .trim(); |
There was a problem hiding this comment.
These aren't valid: parenthesized, any token other than Equal, nested
dhruvmanila
force-pushed
the
dhruv/assign-line-magic
branch
from
a5b96bb to
67ee8a4
Compare
This reverts commit 0a909cc. This will be added in the next PR.
Cloning is expensive especially for every token.
dhruvmanila
force-pushed
the
dhruv/assign-line-magic
branch
from
67ee8a4 to
3dfb8f4
Compare
Emit
MagicCommandtoken when it is the assignment value1 i.e., on the right side of an assignment statement.Examples:
Footnotes
Only
%and!are valid in that position, other magic kinds are not valid ↩