Code

[WIP] Introduce Multi-lingual Support#1

Code

Draft

beme248 wants to merge 227 commits intomainbeme248/datatrove:mainfrom multilingualbeme248/datatrove:multilingualCopy head branch name to clipboard

+1,062,102-694

Commits

Commits on Feb 12, 2024

• Wiki-language statistics + example

  vsabolcec

  committed
  77cf624

  Copy full SHA for 77cf624

• Initial multilingual Gopher quality filter + example

  vsabolcec

  committed
  4134451

  Copy full SHA for 4134451

Commits on Feb 13, 2024

• HuggingFace token via env

  vsabolcec

  committed
  2d0958e

  Copy full SHA for 2d0958e

• Rework process_fineweb.py example

  vsabolcec

  committed
  63d2a7a

  Copy full SHA for 63d2a7a

• Datatrove language stats

  vsabolcec

  committed
  6d0cd05

  Copy full SHA for 6d0cd05

• Rework examples to new pipeline blocks

  vsabolcec

  committed
  23ebd80

  Copy full SHA for 23ebd80

• Initial README for multilingual preprocessing

  vsabolcec

  committed
  17e3b90

  Copy full SHA for 17e3b90

• Slurm example update

  vsabolcec

  committed
  90f8ad8

  Copy full SHA for 90f8ad8

• README update

  vsabolcec

  committed
  2a71ad1

  Copy full SHA for 2a71ad1

Commits on Feb 14, 2024

• Update examples and lang_stats.json

  vsabolcec

  committed
  fae2b05

  Copy full SHA for fae2b05

• Add exclusion_writer to example

  vsabolcec

  committed
  5f590d2

  Copy full SHA for 5f590d2

• Update README

  vsabolcec

  committed
  e2d8dbf

  Copy full SHA for e2d8dbf

• Update README

  vsabolcec

  committed
  3fd2e9d

  Copy full SHA for 3fd2e9d

Commits on Feb 15, 2024

• Unify tokenization

  vsabolcec

  committed
  ead90bc

  Copy full SHA for ead90bc

• README

  vsabolcec

  committed
  59edccf

  Copy full SHA for 59edccf

Commits on Feb 20, 2024

• Word tokenizer support

  vsabolcec

  committed
  c35edea

  Copy full SHA for c35edea

• Formatting

  vsabolcec

  committed
  8fb2133

  Copy full SHA for 8fb2133

• Add stanza as dependency

  vsabolcec

  committed
  beeca91

  Copy full SHA for beeca91

• Additional language statistics

  vsabolcec

  committed
  ca4dd1d

  Copy full SHA for ca4dd1d

• Formatting

  vsabolcec

  committed
  9746b4a

  Copy full SHA for 9746b4a

Commits on Feb 21, 2024

• Lazy load stanza tokenizer

  vsabolcec

  committed
  bbaa097

  Copy full SHA for bbaa097

• Word length distribution stat

  vsabolcec

  committed
  5279b2d

  Copy full SHA for 5279b2d

• Move q() in scope

  vsabolcec

  committed
  9f2e8f9

  Copy full SHA for 9f2e8f9

Commits on Feb 22, 2024

• Word statistics

  vsabolcec

  committed
  ccc9347

  Copy full SHA for ccc9347

• Typo

  vsabolcec

  committed
  1ccfd8e

  Copy full SHA for 1ccfd8e

• Ignore word case in counter

  vsabolcec

  committed
  d549b51

  Copy full SHA for d549b51

• Update lang_stats and README

  vsabolcec

  committed
  17b2c6c

  Copy full SHA for 17b2c6c

Commits on Feb 26, 2024

• Word counter: optional prune

  vsabolcec

  committed
  572a19e

  Copy full SHA for 572a19e

• Top 10000 words

  vsabolcec

  committed
  74cdba5

  Copy full SHA for 74cdba5

• Additional language stats

  vsabolcec

  committed
  695ce52

  Copy full SHA for 695ce52

• Remove util

  vsabolcec

  committed
  0c7e8d5

  Copy full SHA for 0c7e8d5

• Format

  vsabolcec

  committed
  2391880

  Copy full SHA for 2391880

• Stats

  vsabolcec

  committed
  8e03924

  Copy full SHA for 8e03924

• cc_news stats

  vsabolcec

  committed
  2686a2d

  Copy full SHA for 2686a2d

• Stats

  vsabolcec

  committed
  2d5b7a6

  Copy full SHA for 2d5b7a6

• Word stats to file

  vsabolcec

  committed
  53824b6

  Copy full SHA for 53824b6

• Updated stats

  vsabolcec

  committed
  d10e1d7

  Copy full SHA for d10e1d7

• ShuffledHFDataset, wiki sanity check

  vsabolcec

  committed
  c2e4ae5

  Copy full SHA for c2e4ae5

• Named parameter

  vsabolcec

  committed
  33e225f

  Copy full SHA for 33e225f

• Division by zero

  vsabolcec

  committed
  61a957b

  Copy full SHA for 61a957b

• Division by zero: different

  vsabolcec

  committed
  e60c2f0

  Copy full SHA for e60c2f0

• Updated stats

  vsabolcec

  committed
  b07fe8b

  Copy full SHA for b07fe8b

Commits on Feb 27, 2024

• Change thresholds

  vsabolcec

  committed
  100ad99

  Copy full SHA for 100ad99

• Update stats

  vsabolcec

  committed
  afdcce6

  Copy full SHA for afdcce6

• Sanity check and updated stats

  vsabolcec

  committed
  bcab477

  Copy full SHA for bcab477

Commits on Feb 28, 2024

• Sanity check update

  vsabolcec

  committed
  ba8ac7c

  Copy full SHA for ba8ac7c

Commits on Feb 29, 2024

• Stats: count UTF-8 bytes

  vsabolcec

  committed
  3d2906f

  Copy full SHA for 3d2906f

• Commoncrawl stats, format

  vsabolcec

  committed
  a7dc867

  Copy full SHA for a7dc867

• Fix typo

  vsabolcec

  committed
  67c49b4

  Copy full SHA for 67c49b4

• Faster doc stats

  vsabolcec

  committed
  451470a

  Copy full SHA for 451470a

Commits on Mar 1, 2024

• Autoformat

  vsabolcec

  committed
  8080148

  Copy full SHA for 8080148

• MultilingualGopherFilter test

  vsabolcec

  committed
  6282cee

  Copy full SHA for 6282cee

• Merge remote-tracking branch 'upstream/main' into multilingual

  vsabolcec

  committed
  b5c208b

  Copy full SHA for b5c208b

• adjust multi-lingual

  

  NXz64Fdf8Y

  committed
  858585b

  Copy full SHA for 858585b

Commits on Mar 4, 2024

• bugfix mailuser

  guipenedo

  committed
  ff23b94

  Copy full SHA for ff23b94

Commits on Mar 5, 2024

• Add jobs_status command. (huggingface#113)

  Show description for 6baacca
  

  lvwerra

  and

  guipenedo

  authored
  6baacca

  Copy full SHA for 6baacca

• bugfix recursive

  guipenedo

  committed
  366c4cc

  Copy full SHA for 366c4cc

• nit

  guipenedo

  committed
  df0e324

  Copy full SHA for df0e324

• bugfix tokenization unshuffled cleanup

  guipenedo

  committed
  7dbf8ec

  Copy full SHA for 7dbf8ec

• Re-enable datasets test (huggingface#114)

  mariosasko

  authored
  6a15cab

  Copy full SHA for 6a15cab

• nltk stats

  

  NXz64Fdf8Y

  committed
  486acf0

  Copy full SHA for 486acf0

• Merge branch 'multilingual' into bm/run-pipelines

  

  NXz64Fdf8Y

  committed
  fe391a7

  Copy full SHA for fe391a7

Commits on Mar 6, 2024

• Update warc.py (huggingface#115)

  Show description for e1f9524
  jordane95

  authored
  e1f9524

  Copy full SHA for e1f9524

Commits on Mar 7, 2024

• Merge remote-tracking branch 'upstream/main' into multilingual

  vsabolcec

  committed
  97090c5

  Copy full SHA for 97090c5

Commits on Mar 8, 2024

• bugfix doc_len and doc_len_tokens means and std_dev

  guipenedo

  committed
  a98aafd

  Copy full SHA for a98aafd

Commits on Mar 11, 2024

• Add std to language stats

  vsabolcec

  committed
  ba34a3b

  Copy full SHA for ba34a3b

• Update language stats

  vsabolcec

  committed
  8027b16

  Copy full SHA for 8027b16

• Clean-up sanity_check_wiki.py

  vsabolcec

  committed
  eee0935

  Copy full SHA for eee0935

• max_non_alpha_words_ratio per language filter

  vsabolcec

  committed
  9086eff

  Copy full SHA for 9086eff

• Fix typo

  vsabolcec

  committed
  4966a6a

  Copy full SHA for 4966a6a

• Fix no DocumentStats

  vsabolcec

  committed
  6809caa

  Copy full SHA for 6809caa

• Sanity check: min. 1 word

  vsabolcec

  committed
  c66052d

  Copy full SHA for c66052d

Commits on Mar 12, 2024

• Add new languages

  vsabolcec

  committed
  a7ab410

  Copy full SHA for a7ab410

• Add word tokenizers

  vsabolcec

  committed
  72c0cb8

  Copy full SHA for 72c0cb8

Commits on Mar 13, 2024

• Top 50 language stats

  vsabolcec

  committed
  6539ca6

  Copy full SHA for 6539ca6

• Add more word tokenizers

  vsabolcec

  committed
  03272f1

  Copy full SHA for 03272f1

• Change LanguageStatsReducer behaviour

  vsabolcec

  committed
  9d3e505

  Copy full SHA for 9d3e505

• Add LanguageStatsReducer default reduce

  vsabolcec

  committed
  f059380

  Copy full SHA for f059380

• max_non_alpha_words_ratio in language statistics

  vsabolcec

  committed
  20b4b44

  Copy full SHA for 20b4b44

• Update language stats

  vsabolcec

  committed
  f7ffa27

  Copy full SHA for f7ffa27

• draft

  

  NXz64Fdf8Y

  committed
  6ded809

  Copy full SHA for 6ded809

• Merge branch 'multilingual' into bm/run-pipelines

  

  NXz64Fdf8Y

  committed
  0cecc70

  Copy full SHA for 0cecc70

• backup

  

  NXz64Fdf8Y

  committed
  505dcac

  Copy full SHA for 505dcac

• Update language stats

  vsabolcec

  committed
  8b78489

  Copy full SHA for 8b78489

• add languages

  

  NXz64Fdf8Y

  committed
  5ab7f19

  Copy full SHA for 5ab7f19

• Merge branch 'multilingual' into bm/run-pipelines

  

  NXz64Fdf8Y

  committed
  9bef960

  Copy full SHA for 9bef960

Commits on Mar 14, 2024

• More tokenizers and SpaCy switch

  vsabolcec

  committed
  9d0bc00

  Copy full SHA for 9d0bc00

• Add tokenizers, update stat script

  vsabolcec

  committed
  38814ae

  Copy full SHA for 38814ae

• Revert some tokenizers to stanza

  vsabolcec

  committed
  9399c7c

  Copy full SHA for 9399c7c

• Tweak SpaCy

  vsabolcec

  committed
  c721385

  Copy full SHA for c721385

• 95 tokenizers + test

  vsabolcec

  committed
  5fa4927

  Copy full SHA for 5fa4927

• Format code

  vsabolcec

  committed
  47152be

  Copy full SHA for 47152be

• Update language stats

  vsabolcec

  committed
  18a5441

  Copy full SHA for 18a5441

• Top 100 language stats

  vsabolcec

  committed
  37a84f2

  Copy full SHA for 37a84f2

• draft 2

  

  NXz64Fdf8Y

  committed
  5c9e4d5

  Copy full SHA for 5c9e4d5

• draft 3

  

  NXz64Fdf8Y

  committed
  7444a56

  Copy full SHA for 7444a56

Commits on Mar 15, 2024

• Spacy tokenizer ignore whitespaces

  vsabolcec

  committed
  4a797f1

  Copy full SHA for 4a797f1

• Format code

  vsabolcec

  committed
  23c0e66

  Copy full SHA for 23c0e66

• draft 5

  

  NXz64Fdf8Y

  committed
  e2f5096

  Copy full SHA for e2f5096

• Update language stats

  vsabolcec

  committed
  db248b6

  Copy full SHA for db248b6

• Merge branch 'multilingual' into bm/run-pipelines

  

  NXz64Fdf8Y

  committed
  abdae22

  Copy full SHA for abdae22

• Update language stats for top 100 languages

  vsabolcec

  committed
  0a54ba6

  Copy full SHA for 0a54ba6

• draft 6

  

  NXz64Fdf8Y

  committed
  705e412

  Copy full SHA for 705e412

• Merge branch 'multilingual' into bm/run-pipelines

  

  NXz64Fdf8Y

  committed
  8e0fa3d

  Copy full SHA for 8e0fa3d

• IndicNLP tokenizer fix

  vsabolcec

  committed
  aadb17f

  Copy full SHA for aadb17f

• darft 9

  

  NXz64Fdf8Y

  committed
  fc229cf

  Copy full SHA for fc229cf

• Merge branch 'multilingual' into bm/run-pipelines

  

  NXz64Fdf8Y

  committed
  3a00b5a

  Copy full SHA for 3a00b5a

• Word tokenizer test

  vsabolcec

  committed
  f6ec925

  Copy full SHA for f6ec925

• run spacy v2 top100

  

  NXz64Fdf8Y

  committed
  895b0e8

  Copy full SHA for 895b0e8

• Bug fix: when file is empty (huggingface#126)

  Show description for 6d708bd
  jordane95

  authored
  6d708bd

  Copy full SHA for 6d708bd

Commits on Mar 16, 2024

• fix

  

  NXz64Fdf8Y

  committed
  f2a0ee0

  Copy full SHA for f2a0ee0

• add aggregated stats

  

  NXz64Fdf8Y

  committed
  ea7c10b

  Copy full SHA for ea7c10b

Commits on Mar 17, 2024

• update stats

  

  NXz64Fdf8Y

  committed
  c795863

  Copy full SHA for c795863

Commits on Mar 18, 2024

• Korean and Thai faster tokenizers

  vsabolcec

  committed
  69bbbb8

  Copy full SHA for 69bbbb8

• Split language statistics into multiple files

  vsabolcec

  committed
  21d0607

  Copy full SHA for 21d0607

Commits on Mar 19, 2024

• Split language statistics into multiple files (2)

  vsabolcec

  committed
  b9bd9fb

  Copy full SHA for b9bd9fb

• Separate filter parameters

  vsabolcec

  committed
  f5655cc

  Copy full SHA for f5655cc

• Remove testing language and format code

  vsabolcec

  committed
  5e673b6

  Copy full SHA for 5e673b6

• Introduce LanguageStats dataclass and clean up

  vsabolcec

  committed
  b2020a4

  Copy full SHA for b2020a4

• Remove alternative lang_stats scripts

  vsabolcec

  committed
  0bca5a8

  Copy full SHA for 0bca5a8

• LanguageStatsReducer output to yml

  vsabolcec

  committed
  52ee2c5

  Copy full SHA for 52ee2c5

• Load tokenizer using from_file (huggingface#122)

  Show description for b98ef1c
  guipenedo

  authored
  b98ef1c

  Copy full SHA for b98ef1c

• add depends= to LocalPipelineExecutor (huggingface#100)

  guipenedo

  authored
  bc57162

  Copy full SHA for bc57162

• Remove commoncrawl_stats.py

  vsabolcec

  committed
  35fc009

  Copy full SHA for 35fc009

• Update multilingual README.md

  vsabolcec

  committed
  04fb13f

  Copy full SHA for 04fb13f

• Revert remove most_common(10000)

  vsabolcec

  committed
  494d0a7

  Copy full SHA for 494d0a7

• Convert .json language stats into .yml

  vsabolcec

  committed
  0ac5745

  Copy full SHA for 0ac5745

• Merge pull request #5 from beme248/bm/run-pipelines

  Show description for 1d63b82
  vsabolcec

  authored
  1d63b82

  Copy full SHA for 1d63b82

• Combine branches and use .yml in process_non_english.py

  vsabolcec

  committed
  2a4aedb

  Copy full SHA for 2a4aedb

Commits on Mar 20, 2024

• Wiki language stats RUN_MODE

  vsabolcec

  committed
  644ca12

  Copy full SHA for 644ca12

• Remove testing part from Wiki language stats script

  vsabolcec

  committed
  e74101a

  Copy full SHA for e74101a

• Update README

  vsabolcec

  committed
  349fdb1

  Copy full SHA for 349fdb1

• Clean up

  vsabolcec

  committed
  ba63075

  Copy full SHA for ba63075

• Improve C4 filter and dedup (huggingface#124)

  Show description for 55c6b1c
  guipenedo

  authored
  55c6b1c

  Copy full SHA for 55c6b1c

• Adds option to shuffle input files in readers (huggingface#128)

  Show description for 27e2cea
  guipenedo

  authored
  27e2cea

  Copy full SHA for 27e2cea

• update Trafilatura version (huggingface#130)

  adbar

  authored
  b6cd366

  Copy full SHA for b6cd366

Commits on Mar 21, 2024

• Update README

  vsabolcec

  committed
  8355059

  Copy full SHA for 8355059

• Typo

  vsabolcec

  committed
  304d34c

  Copy full SHA for 304d34c

Commits on Mar 22, 2024

• Changes to text normalization + FTFY and lines symbol formatters (huggingface#133)

  Show description for 6daa5e8
  guipenedo

  authored
  6daa5e8

  Copy full SHA for 6daa5e8

• removed debug print

  guipenedo

  committed
  8421fe1

  Copy full SHA for 8421fe1

Commits on Mar 23, 2024

• fix symbollinesremover regex hanging

  guipenedo

  committed
  87e30a4

  Copy full SHA for 87e30a4

• Minor Terminology and Documentation Updates for Local Tokenizer Loading (huggingface#134)

  Show description for 56aa210
  justHungryMan

  authored
  56aa210

  Copy full SHA for 56aa210

Commits on Mar 27, 2024

• Tweak process_non_english.py

  vsabolcec

  committed
  c2d61de

  Copy full SHA for c2d61de

Commits on Apr 2, 2024

• Merge remote-tracking branch 'upstream/main' into pretokenization

  vsabolcec

  committed
  d44a51c

  Copy full SHA for d44a51c

• Add tokenizers

  vsabolcec

  committed
  5c6f1bc

  Copy full SHA for 5c6f1bc

• Type fix

  vsabolcec

  committed
  c4d6fce

  Copy full SHA for c4d6fce

• add requeue and QOS slurm options (huggingface#144)

  Show description for afadc8f
  

  3 peopleauthored
  afadc8f

  Copy full SHA for afadc8f

Commits on Apr 3, 2024

• Merge branch 'huggingface:main' into pretokenization

  vsabolcec

  authored
  944fb21

  Copy full SHA for 944fb21

• English- and Korean-only tokenization

  vsabolcec

  committed
  cca0e41

  Copy full SHA for cca0e41

• English tokenizer test

  vsabolcec

  committed
  5933335

  Copy full SHA for 5933335

Commits on Apr 5, 2024

• Fix substring dedup range (huggingface#132)

  Show description for 670fc40
  jordane95

  authored
  670fc40

  Copy full SHA for 670fc40

• Line dedup min remove words option (huggingface#146)

  Show description for 8c7e052
  guipenedo

  authored
  8c7e052

  Copy full SHA for 8c7e052

Commits on Apr 10, 2024

• Add multilang tokenizer to Gopher quality filter

  vsabolcec

  committed
  22cba4c

  Copy full SHA for 22cba4c

• Require language metadata in Gopher quality

  vsabolcec

  committed
  575d98f

  Copy full SHA for 575d98f

• Merge branch 'huggingface:main' into pretokenization

  vsabolcec

  authored
  48377af

  Copy full SHA for 48377af

• Lazy-load and separate dependencies

  vsabolcec

  committed
  1e8c375

  Copy full SHA for 1e8c375

• fix timeout related issues in extractors

  guipenedo

  committed
  5015a4c

  Copy full SHA for 5015a4c

Commits on Apr 11, 2024

• Move top-level import to classes

  vsabolcec

  committed
  e6b1ccf

  Copy full SHA for e6b1ccf

• Remove print in tests

  vsabolcec

  committed
  93d5c2a

  Copy full SHA for 93d5c2a

• pyproject.toml: tokenization -> multilingual

  vsabolcec

  committed
  005a3ef

  Copy full SHA for 005a3ef

• New options for FastTextClassifierFilter: apply on sentence or paragraph (line) level (huggingface#151)

  Show description for 40a3a1b
  guipenedo

  authored
  40a3a1b

  Copy full SHA for 40a3a1b

Commits on Apr 12, 2024

• Url deduplication (huggingface#145)

  Show description for 209ebec
  

  hynky1999

  and

  guipenedo

  authored
  209ebec

  Copy full SHA for 209ebec

Commits on Apr 15, 2024

• fetch all the labels on fasttextfilter

  guipenedo

  committed
  01175ba

  Copy full SHA for 01175ba

• add min_num_sentences to line dedup

  guipenedo

  committed
  854389f

  Copy full SHA for 854389f

Commits on Apr 16, 2024

• fix tests

  guipenedo

  committed
  6614f06

  Copy full SHA for 6614f06

• Fix race conditions during download/extraction (huggingface#155)

  Show description for 7a0f6c4
  

  3 peopleauthored
  7a0f6c4

  Copy full SHA for 7a0f6c4

• Adds PII removal (huggingface#156)

  Show description for 9d75443
  

  3 peopleauthored
  9d75443

  Copy full SHA for 9d75443

Commits on Apr 20, 2024

• fineweb filter

  hynky1999

  committed
  2aa32d4

  Copy full SHA for 2aa32d4

• fw filters

  hynky1999

  committed
  c4ee193

  Copy full SHA for c4ee193

• remove duplicate filter

  hynky1999

  committed
  9a88beb

  Copy full SHA for 9a88beb

• clean up + fineweb example

  guipenedo

  committed
  05194d3

  Copy full SHA for 05194d3

• added PII block

  guipenedo

  committed
  447c942

  Copy full SHA for 447c942

Commits on Apr 22, 2024

• Pypi Publish Action (huggingface#159)

  Show description for 50eb055
  hynky1999

  authored
  50eb055

  Copy full SHA for 50eb055

• Update pyproject.toml

  guipenedo

  authored
  1724f28

  Copy full SHA for 1724f28

• Update pypi-release.yml

  guipenedo

  authored
  7042785

  Copy full SHA for 7042785

• Update pypi-release.yml

  guipenedo

  authored
  6d06210

  Copy full SHA for 6d06210

Commits on Apr 24, 2024

• Added c4 badwords filter, added batch tokenization to tokenscounter (huggingface#160)

  Show description for 4e9235f
  guipenedo

  authored
  4e9235f

  Copy full SHA for 4e9235f

Commits on Apr 26, 2024

• fix documents with a lot of paragraphs being removed by the repetition filter

  guipenedo

  committed
  a8d21e2

  Copy full SHA for a8d21e2

• FineWeb quality statistics

  vsabolcec

  committed
  d3f8245

  Copy full SHA for d3f8245

Commits on Apr 27, 2024

• Merge upstream/main

  vsabolcec

  committed
  1ba51c3

  Copy full SHA for 1ba51c3

• Multilang tokenizer for MinHash

  vsabolcec

  committed
  83077f1

  Copy full SHA for 83077f1

• Remove print

  vsabolcec

  committed
  ae8b4de

  Copy full SHA for ae8b4de

Commits on Apr 28, 2024

• Change Korean Tokenizer with Kiwi

  Kesta-bos

  committed
  0c7e54a

  Copy full SHA for 0c7e54a

• fix

  Kesta-bos

  committed
  cd1587c

  Copy full SHA for cd1587c

• add kiwipiepy dependency

  Kesta-bos

  committed
  a687ef0

  Copy full SHA for a687ef0

Commits on May 2, 2024

• bugfix pii emails and quality filters default args

  guipenedo

  committed
  c72b1e4

  Copy full SHA for c72b1e4

Commits on May 3, 2024

• Add a skip parameter to all readers (defaults to zero) (huggingface#167)

  Show description for 6a4881d
  rantav

  authored
  6a4881d

  Copy full SHA for 6a4881d

Commits on May 4, 2024

• Adds n-gram based decontamination (huggingface#172)

  Show description for 15c8425
  guipenedo

  authored
  15c8425

  Copy full SHA for 15c8425

• add skip in decont index builder

  guipenedo

  committed
  d56d3c5

  Copy full SHA for d56d3c5

Commits on May 6, 2024

• Add repetition stats and update wiki script

  vsabolcec

  committed
  b8ce9c1

  Copy full SHA for b8ce9c1

• fix for requeueing code and change minhash default

  guipenedo

  committed
  22c739e

  Copy full SHA for 22c739e

Commits on May 7, 2024

• Update stats

  vsabolcec

  committed
  750f0fe

  Copy full SHA for 750f0fe

• Fix division by zero

  vsabolcec

  committed
  af647a7

  Copy full SHA for af647a7

• Handle non-method cases in to_dict conversion (huggingface#139)

  justHungryMan

  authored
  8805e13

  Copy full SHA for 8805e13

• Adds tasks_per_job to slurm executor (huggingface#153)

  Show description for ff50473
  guipenedo

  authored
  ff50473

  Copy full SHA for ff50473

• Update statistics

  vsabolcec

  committed
  12f79a1

  Copy full SHA for 12f79a1

• Unsigned int tokenizer and srun args (huggingface#154)

  Show description for b2b96e4
  

  3 peopleauthored
  b2b96e4

  Copy full SHA for b2b96e4

• Update filters

  vsabolcec

  committed
  caeca43

  Copy full SHA for caeca43

• add linting to the examples folder

  guipenedo

  committed
  a970249

  Copy full SHA for a970249

• Enhance BaseReader to allow custom adapters access to instance variables (huggingface#169)

  Show description for 4d83342
  

  3 peopleauthored
  4d83342

  Copy full SHA for 4d83342

Commits on May 8, 2024

• Merge branch 'beme248:multilingual' into multilingual

  Kesta-bos

  authored
  b246107

  Copy full SHA for b246107

• Merge pull request #6 from Kesta-bos/multilingual

  Show description for 4218ec8
  vsabolcec

  authored
  4218ec8

  Copy full SHA for 4218ec8

• Format

  vsabolcec

  committed
  9ef64e8

  Copy full SHA for 9ef64e8

• Merge branch 'huggingface:main' into multilingual

  vsabolcec

  authored
  a6eca69

  Copy full SHA for a6eca69

• Fix Shuffled HFReader

  vsabolcec

  committed
  ed5244a

  Copy full SHA for ed5244a

• Update Korean stats and filters

  vsabolcec

  committed
  4072131

  Copy full SHA for 4072131

Commits on May 12, 2024

• Better FineWeb

  vsabolcec

  committed
  abd1534

  Copy full SHA for abd1534

• Update statistics

  vsabolcec

  committed
  331c439

  Copy full SHA for 331c439

• Format

  vsabolcec

  committed
  220e235

  Copy full SHA for 220e235

Commits on May 13, 2024

• MultilingualFineWebQualityFilter

  vsabolcec

  committed
  87720e5

  Copy full SHA for 87720e5

• Merge origin/pretokenization

  vsabolcec

  committed
  9e4b222

  Copy full SHA for 9e4b222

• Lazy load and tokenize -> word_tokenize

  vsabolcec

  committed
  7bd1093

  Copy full SHA for 7bd1093

• sent_tokenize

  vsabolcec

  committed
  ebb8f0f

  Copy full SHA for ebb8f0f

• Move tokenization dependencies to multilingual

  vsabolcec

  committed
  4b97552

  Copy full SHA for 4b97552

• C4 quality sent_tokenize

  vsabolcec

  committed
  a92222a

  Copy full SHA for a92222a

• Langstats tokenizer fix

  vsabolcec

  committed
  19e04d8

  Copy full SHA for 19e04d8

• Langstats tokenizer fix 2

  vsabolcec

  committed
  bcac6c5

  Copy full SHA for bcac6c5

• Sent tokenize tweaks

  vsabolcec

  committed
  545d379

  Copy full SHA for 545d379

• Whitespace stripping

  vsabolcec

  committed
  7ae2fa7

  Copy full SHA for 7ae2fa7

• Fallback tokenizer

  vsabolcec

  committed
  7034978

  Copy full SHA for 7034978

Commits on May 14, 2024

• Khmer punctuation

  vsabolcec

  committed
  b2ae282

  Copy full SHA for b2ae282

• Update Wiki stats script

  vsabolcec

  committed
  5846fbd

  Copy full SHA for 5846fbd

• Format

  vsabolcec

  committed
  c0505de

  Copy full SHA for c0505de

• Update stat and filter #1

  vsabolcec

  committed
  e3936fb

  Copy full SHA for e3936fb

• Remove nlpashto

  vsabolcec

  committed
  2c61271

  Copy full SHA for 2c61271

• Update stats and filters #2

  vsabolcec

  committed
  4f5d07e

  Copy full SHA for 4f5d07e

Commits on May 28, 2024

• Small readme fix

  vsabolcec

  committed
  28b0205

  Copy full SHA for 28b0205