Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

better tokenization #162

Merged
merged 23 commits into from
Mar 18, 2021
Merged

better tokenization #162

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
1592f0e
fix/literal: cover /*, /**, and /*!
drahnr Feb 26, 2021
6a949e3
fix/blockcomment: additional tests and logic for block comments
drahnr Feb 26, 2021
e56aa25
chore: cargo fmt
drahnr Feb 26, 2021
e766459
ci: make timeouts more generous to not fail builds on initial runs
drahnr Mar 17, 2021
eeab31e
silly attempt to improve tokenization
drahnr Mar 17, 2021
1dba163
chore: bump deps bitflags + ra
drahnr Mar 17, 2021
c2d29c9
add extra tokenization split chars option
drahnr Mar 17, 2021
44999a2
drop deprecated languagetool backend
drahnr Mar 17, 2021
4bfd600
remove unused srx crate
drahnr Mar 17, 2021
e47404f
always tokenize based on nlprule backend
drahnr Mar 17, 2021
63639da
new tokenizer also keeps marks
drahnr Mar 17, 2021
ee62aa3
chore: cargo update
drahnr Mar 17, 2021
ec7d74c
chore: cargo fmt
drahnr Mar 17, 2021
49d1d0b
fix: rebase fallout
drahnr Mar 18, 2021
80e4d91
feat: add --jobs option and improve a few bug on messages
drahnr Mar 18, 2021
e7202b2
fix: adjust byte offset for new tokenizer
drahnr Mar 18, 2021
4378bcd
refactor: log messages about thread count / jobs
drahnr Mar 18, 2021
1d0d102
refactor/args: add logs
drahnr Mar 18, 2021
7a3117d
chore: cargo format use fs_err where possible
drahnr Mar 18, 2021
03a8f85
fix/tokenization: ignore known sentence ctrl characters
drahnr Mar 18, 2021
bb4d7f6
better logging for traversal
drahnr Mar 18, 2021
27bd2ba
fix/traverse: make sure all modules are found
drahnr Mar 18, 2021
7bda5a4
chore: cargo fmt
drahnr Mar 18, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .concourse.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
- name: pr-validate
build_logs_to_retain: 10
public: true
serial: false
serial: true
plan:
- in_parallel:
- get: git-pull-request-resource
Expand Down Expand Up @@ -114,7 +114,7 @@ jobs:

- in_parallel:
- task: compile-pr
timeout: 10m
timeout: 20m
image: env-glibc
config:
platform: linux
Expand Down Expand Up @@ -159,7 +159,7 @@ jobs:
status: failure

- task: unit-tests-pr
timeout: 10m
timeout: 20m
image: env-glibc
config:
platform: linux
Expand Down Expand Up @@ -193,7 +193,7 @@ jobs:
status: failure

- task: pr-win64-cross-build
timeout: 15m
timeout: 20m
image: env-glibc
config:
platform: linux
Expand Down
Loading