Code

Trainer clean up#52

Code

Merged

XapaJIaMnu merged 43 commits intotrainerhplt-project/OpusCleaner:trainerfrom trainer-clean-uphplt-project/OpusCleaner:trainer-clean-upCopy head branch name to clipboardDec 12, 2022

+1,038-373

Commits

Commits on Nov 21, 2022

• Reorganise trainer to be more explicit about state

  Show description for cb81e26
  jelmervdl

  committed
  cb81e26

  Copy full SHA for cb81e26

Commits on Nov 28, 2022

• Fix bugs

  jelmervdl

  committed
  4829efd

  Copy full SHA for 4829efd

• V1 and V2 curriculum yaml structures

  jelmervdl

  committed
  e2f26de

  Copy full SHA for e2f26de

• Share readers across all stages

  jelmervdl

  committed
  8b39c41

  Copy full SHA for 8b39c41

• Don't dump state after every batch

  Show description for e2983b7
  jelmervdl

  committed
  e2983b7

  Copy full SHA for e2983b7

• Add documentation & safety checks

  jelmervdl

  committed
  5d21c12

  Copy full SHA for 5d21c12

• Add async shuffling

  Show description for 60c92be
  jelmervdl

  committed
  60c92be

  Copy full SHA for 60c92be

• Discard pre-shuffled data when restoring

  jelmervdl

  committed
  a010e36

  Copy full SHA for a010e36

• Properly clean up resources if close() is used

  jelmervdl

  committed
  0cb7478

  Copy full SHA for 0cb7478

• Start adding unit tests

  Show description for 79b1769
  jelmervdl

  committed
  79b1769

  Copy full SHA for 79b1769

• Move trainer Popen call out of trainer, add unit test for state resumption

  jelmervdl

  committed
  1dd42b0

  Copy full SHA for 1dd42b0

• Naming things

  jelmervdl

  committed
  ab288e0

  Copy full SHA for ab288e0

Commits on Nov 29, 2022

• Replace default shuffler with one that does file-based shuffling

  jelmervdl

  committed
  983c22e

  Copy full SHA for 983c22e

Commits on Dec 2, 2022

• Remove random.sh! Why? Click here to find out!

  Show description for 45dcdb9
  jelmervdl

  committed
  45dcdb9

  Copy full SHA for 45dcdb9

• You will not believe what happened to shuffle.py!

  Show description for 7af737c
  jelmervdl

  committed
  7af737c

  Copy full SHA for 7af737c

Commits on Dec 4, 2022

• Add deduplication functionality to shuffle.py

  jelmervdl

  committed
  420d437

  Copy full SHA for 420d437

• Fix modifiers messing up newlines

  jelmervdl

  committed
  2beac2f

  Copy full SHA for 2beac2f

• Deduplicate trainer input, and add flag to flip src/trg sentence side

  jelmervdl

  committed
  33f9257

  Copy full SHA for 33f9257

• Make file path relative to configuration file

  jelmervdl

  committed
  8e6d561

  Copy full SHA for 8e6d561

Commits on Dec 5, 2022

• Allow actual training program to be specified on the cli.

  Show description for cd4cbbc
  jelmervdl

  committed
  cd4cbbc

  Copy full SHA for cd4cbbc

• split -> split_into_tempfiles

  jelmervdl

  committed
  863ca73

  Copy full SHA for 863ca73

• Add explicit option for setting the temporary directory

  jelmervdl

  committed
  5bb3fb7

  Copy full SHA for 5bb3fb7

• Add comment as to why _read_gzip is implemented that way

  jelmervdl

  committed
  701f7fd

  Copy full SHA for 701f7fd

• Remove oops

  jelmervdl

  committed
  f048fd4

  Copy full SHA for f048fd4

• Document DatasetReader some more

  jelmervdl

  committed
  e820a7d

  Copy full SHA for e820a7d

• Rename dataset_name to until_dataset_name for that last dataset

  jelmervdl

  committed
  d8c1cc7

  Copy full SHA for d8c1cc7

• Formalise & document disabling state loading

  jelmervdl

  committed
  d1a03df

  Copy full SHA for d1a03df

Commits on Dec 6, 2022

• Rework threading in shuffler

  Show description for 1b09891
  jelmervdl

  committed
  1b09891

  Copy full SHA for 1b09891

• Fix bugs

  jelmervdl

  committed
  120ce04

  Copy full SHA for 120ce04

• Add quick sanity check as to whether all dataset files exist

  jelmervdl

  committed
  ca3a8e1

  Copy full SHA for ca3a8e1

• exist does not exist but exists does

  jelmervdl

  committed
  22f3534

  Copy full SHA for 22f3534

• Fix missing import

  Show description for 812723a
  jelmervdl

  committed
  812723a

  Copy full SHA for 812723a

Commits on Dec 7, 2022

• Closer control the marian subprocess

  jelmervdl

  committed
  fddb7fb

  Copy full SHA for fddb7fb

• Make the trainer a bit more verbose, telling us what it is doing

  Show description for 5dd932c
  jelmervdl

  committed
  5dd932c

  Copy full SHA for 5dd932c

• We're sending 15, not 9

  jelmervdl

  committed
  c5e1726

  Copy full SHA for c5e1726

• Add f prefixes I missed

  jelmervdl

  committed
  bc35c90

  Copy full SHA for bc35c90

Commits on Dec 8, 2022

• Remove flip option

  jelmervdl

  committed
  4a82c82

  Copy full SHA for 4a82c82

• Remove deduplication from shuffle.py

  jelmervdl

  committed
  590f649

  Copy full SHA for 590f649

• Remove bufsize=0 as it is not clear whether this is an improvement

  jelmervdl

  committed
  73ccd0b

  Copy full SHA for 73ccd0b

• Add third ctrl-c for force-quit trainer

  jelmervdl

  committed
  46db743

  Copy full SHA for 46db743

• Change configuration format

  Show description for 51f7212
  jelmervdl

  committed
  51f7212

  Copy full SHA for 51f7212

• Update & document tests

  jelmervdl

  committed
  bf4df4c

  Copy full SHA for bf4df4c

Commits on Dec 12, 2022

• Update README.md

  jelmervdl

  authored
  e0c2326

  Copy full SHA for e0c2326