feat: add length and time limit to tokenizer #588
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for shiki-next ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
✅ Deploy Preview for shiki-matsu ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #588 +/- ##
==========================================
+ Coverage 96.15% 96.17% +0.02%
==========================================
Files 70 70
Lines 5667 5698 +31
Branches 749 751 +2
==========================================
+ Hits 5449 5480 +31
Misses 212 212
Partials 6 6 ☔ View full report in Codecov by Sentry. |
ije
changed the title
timeLimit parameter of tokenizeLine2 to 500mstimeLimit parameter of tokenizeLine2 to 500ms
ije
changed the title
timeLimit parameter of tokenizeLine2 to 500ms
ije
changed the title
antfu
reviewed
Feb 15, 2024
| @@ -59,6 +59,19 @@ export function tokenizeWithTheme( | |||
| continue | |||
| } | |||
|
|
|||
| // Do not attempt to tokenize if a line is too long | |||
| const maxTokenizationLineLength = 20000 | |||
There was a problem hiding this comment.
I am not sure about this. It makes sense to do it in Monaco, but for Shiki core, I think the users are responsible for controlling what passes in. We shouldn't disallow ppl to do so if they want deliberately.
There was a problem hiding this comment.
do you like an option? (default is disable)
There was a problem hiding this comment.
done, please check the testing.
antfu
changed the title
diegohaz
referenced
this pull request
in ariakit/ariakit
Feb 23, 2024
[](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [shiki](https://togithub.com/shikijs/shiki) ([source](https://togithub.com/shikijs/shiki/tree/HEAD/packages/shiki)) | [`1.1.1` -> `1.1.6`](https://renovatebot.com/diffs/npm/shiki/1.1.1/1.1.6) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | [](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>shikijs/shiki (shiki)</summary> ### [`v1.1.6`](https://togithub.com/shikijs/shiki/releases/tag/v1.1.6) [Compare Source](https://togithub.com/shikijs/shiki/compare/v1.1.5...v1.1.6) ##### 🚀 Features - Provide lang sub module for alias - by [@​antfu](https://togithub.com/antfu) in [https://github.com/shikijs/shiki/issues/596](https://togithub.com/shikijs/shiki/issues/596) [<samp>(b61a6)</samp>](https://togithub.com/shikijs/shiki/commit/b61a689b) - Include new `Typst` grammar - by [@​antfu](https://togithub.com/antfu) [<samp>(9f70e)</samp>](https://togithub.com/shikijs/shiki/commit/9f70eed4) ##### 🐞 Bug Fixes - **vitepress**: Missing `:shown` prop for query, fix [#​595](https://togithub.com/shikijs/shiki/issues/595) - by [@​antfu](https://togithub.com/antfu) in [https://github.com/shikijs/shiki/issues/595](https://togithub.com/shikijs/shiki/issues/595) [<samp>(b667b)</samp>](https://togithub.com/shikijs/shiki/commit/b667b75a) ##### [View changes on GitHub](https://togithub.com/shikijs/shiki/compare/v1.1.5...v1.1.6) ### [`v1.1.5`](https://togithub.com/shikijs/shiki/releases/tag/v1.1.5) [Compare Source](https://togithub.com/shikijs/shiki/compare/v1.1.4...v1.1.5) ##### 🐞 Bug Fixes - **monaco**: Editor colors interop, fix [#​594](https://togithub.com/shikijs/shiki/issues/594) - by [@​antfu](https://togithub.com/antfu) in [https://github.com/shikijs/shiki/issues/594](https://togithub.com/shikijs/shiki/issues/594) [<samp>(78825)</samp>](https://togithub.com/shikijs/shiki/commit/788250d7) ##### [View changes on GitHub](https://togithub.com/shikijs/shiki/compare/v1.1.4...v1.1.5) ### [`v1.1.4`](https://togithub.com/shikijs/shiki/releases/tag/v1.1.4) [Compare Source](https://togithub.com/shikijs/shiki/compare/v1.1.3...v1.1.4) ##### 🚀 Features - Add length and time limit to tokenizer - by [@​ije](https://togithub.com/ije) and [@​antfu](https://togithub.com/antfu) in [https://github.com/shikijs/shiki/issues/588](https://togithub.com/shikijs/shiki/issues/588) [<samp>(2803f)</samp>](https://togithub.com/shikijs/shiki/commit/2803f898) ##### 🐞 Bug Fixes - **monaco**: Normalize hex color for monaco, fix [#​594](https://togithub.com/shikijs/shiki/issues/594) - by [@​antfu](https://togithub.com/antfu) in [https://github.com/shikijs/shiki/issues/594](https://togithub.com/shikijs/shiki/issues/594) [<samp>(ecb36)</samp>](https://togithub.com/shikijs/shiki/commit/ecb36e23) ##### [View changes on GitHub](https://togithub.com/shikijs/shiki/compare/v1.1.3...v1.1.4) ### [`v1.1.3`](https://togithub.com/shikijs/shiki/releases/tag/v1.1.3) [Compare Source](https://togithub.com/shikijs/shiki/compare/v1.1.2...v1.1.3) ##### 🚀 Features - New terraform lang, update deps - by [@​antfu](https://togithub.com/antfu) [<samp>(52d0a)</samp>](https://togithub.com/shikijs/shiki/commit/52d0a62d) ##### 🐞 Bug Fixes - **vitepress**: Don't set exit code when throws is set to false - by [@​antfu](https://togithub.com/antfu) [<samp>(e2a0a)</samp>](https://togithub.com/shikijs/shiki/commit/e2a0a432) ##### [View changes on GitHub](https://togithub.com/shikijs/shiki/compare/v1.1.2...v1.1.3) ### [`v1.1.2`](https://togithub.com/shikijs/shiki/releases/tag/v1.1.2) [Compare Source](https://togithub.com/shikijs/shiki/compare/v1.1.1...v1.1.2) ##### 🚀 Features - New lang Move and new theme Vesper - by [@​antfu](https://togithub.com/antfu) [<samp>(8247b)</samp>](https://togithub.com/shikijs/shiki/commit/8247ba72) ##### 🐞 Bug Fixes - Remove trailing newline added by markdown-it - by [@​KermanX](https://togithub.com/KermanX) and [@​antfu](https://togithub.com/antfu) in [https://github.com/shikijs/shiki/issues/585](https://togithub.com/shikijs/shiki/issues/585) [<samp>(b559c)</samp>](https://togithub.com/shikijs/shiki/commit/b559cde5) - **twoslash**: Improve error handling - by [@​antfu](https://togithub.com/antfu) [<samp>(73fdb)</samp>](https://togithub.com/shikijs/shiki/commit/73fdbd17) ##### [View changes on GitHub](https://togithub.com/shikijs/shiki/compare/v1.1.1...v1.1.2) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/ariakit/ariakit). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xNTMuMiIsInVwZGF0ZWRJblZlciI6IjM3LjIwMC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
](https://renovatebot.com){kind=link}
This was referenced May 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fix the textmate tokenizer freeze/crash the browser if a line is too long:
timeLimitparameter oftokenizeLine2to 500ms (align to vscode)tokenizeMaxLineLengthoption)Linked Issues
_
Additional context
Refs
timeLimit:https://github.com/microsoft/vscode/blob/3ce7ccff4116da34370cecfb329420f266499ec4/src/vs/workbench/services/textMate/browser/tokenizationSupport/textMateTokenizationSupport.ts#L54
Do not attempt:
https://github.com/microsoft/vscode/blob/6722b81416d8f1ee24209ca4a19792b6e6a55beb/src/vs/workbench/services/textMate/browser/tokenizationSupport/tokenizationSupportWithLineLimit.ts#L38