Code

better tokenization#162

Code

Merged

drahnr merged 23 commits intomasterdrahnr/cargo-spellcheck:masterfrom bernhard-better-tokenizationdrahnr/cargo-spellcheck:bernhard-better-tokenizationCopy head branch name to clipboardMar 18, 2021

+598-1,768

Commits

Commits on Mar 18, 2021

• fix/literal: cover /*, /**, and /*!

  drahnr

  committed
  1592f0e

  Copy full SHA for 1592f0e

• fix/blockcomment: additional tests and logic for block comments

  drahnr

  committed
  6a949e3

  Copy full SHA for 6a949e3

• chore: cargo fmt

  drahnr

  committed
  e56aa25

  Copy full SHA for e56aa25

• ci: make timeouts more generous to not fail builds on initial runs

  drahnr

  committed
  e766459

  Copy full SHA for e766459

• silly attempt to improve tokenization

  drahnr

  committed
  eeab31e

  Copy full SHA for eeab31e

• chore: bump deps bitflags + ra

  drahnr

  committed
  1dba163

  Copy full SHA for 1dba163

• add extra tokenization split chars option

  drahnr

  committed
  c2d29c9

  Copy full SHA for c2d29c9

• drop deprecated languagetool backend

  drahnr

  committed
  44999a2

  Copy full SHA for 44999a2

• remove unused srx crate

  drahnr

  committed
  4bfd600

  Copy full SHA for 4bfd600

• always tokenize based on nlprule backend

  drahnr

  committed
  e47404f

  Copy full SHA for e47404f

• new tokenizer also keeps marks

  drahnr

  committed
  63639da

  Copy full SHA for 63639da

• chore: cargo update

  drahnr

  committed
  ee62aa3

  Copy full SHA for ee62aa3

• chore: cargo fmt

  drahnr

  committed
  ec7d74c

  Copy full SHA for ec7d74c

• fix: rebase fallout

  drahnr

  committed
  49d1d0b

  Copy full SHA for 49d1d0b

• feat: add --jobs option and improve a few bug on messages

  drahnr

  committed
  80e4d91

  Copy full SHA for 80e4d91

• fix: adjust byte offset for new tokenizer

  drahnr

  committed
  e7202b2

  Copy full SHA for e7202b2

• refactor: log messages about thread count / jobs

  drahnr

  committed
  4378bcd

  Copy full SHA for 4378bcd

• refactor/args: add logs

  drahnr

  committed
  1d0d102

  Copy full SHA for 1d0d102

• chore: cargo format use fs_err where possible

  drahnr

  committed
  7a3117d

  Copy full SHA for 7a3117d

• fix/tokenization: ignore known sentence ctrl characters

  drahnr

  committed
  03a8f85

  Copy full SHA for 03a8f85

• better logging for traversal

  drahnr

  committed
  bb4d7f6

  Copy full SHA for bb4d7f6

• fix/traverse: make sure all modules are found

  Show description for 27bd2ba
  drahnr

  committed
  27bd2ba

  Copy full SHA for 27bd2ba

• chore: cargo fmt

  drahnr

  committed
  7bda5a4

  Copy full SHA for 7bda5a4