Code

Ludwig's evaluation scaffolding#2

Code

Merged

dirkgr merged 26 commits intomainallenai/catwalk:mainfrom Ludwigallenai/catwalk:LudwigCopy head branch name to clipboardMar 3, 2022

+1,380-13

Commits

Commits on Feb 21, 2022

• Ludwig's evaluation scaffolding

  dirkgr

  committed
  deded0d

  Copy full SHA for deded0d

• I didn't mean to check this in.

  dirkgr

  committed
  dba1797

  Copy full SHA for dba1797

Commits on Feb 22, 2022

• Separate running the model and running the evaluation of the results into two different steps

  dirkgr

  committed
  4e6e69f

  Copy full SHA for 4e6e69f

Commits on Feb 23, 2022

• Adds metadata

  dirkgr

  committed
  3ba0ac6

  Copy full SHA for 3ba0ac6

• Optional context for QA

  dirkgr

  committed
  3180e31

  Copy full SHA for 3180e31

• Dataclasses don't support default arguments

  dirkgr

  committed
  cc292bb

  Copy full SHA for cc292bb

• Squad v2 task

  dirkgr

  committed
  446f16b

  Copy full SHA for 446f16b

• Run multiple tasks in one command

  dirkgr

  committed
  0c2fb5f

  Copy full SHA for 0c2fb5f

• Rename model_name to pretrained_model_name_or_path

  dirkgr

  committed
  a4725c8

  Copy full SHA for a4725c8

• Add an option for only running a few instances, for testing

  dirkgr

  committed
  42673ef

  Copy full SHA for 42673ef

Commits on Feb 24, 2022

• We can now depend on Tango@main

  dirkgr

  committed
  06a8f24

  Copy full SHA for 06a8f24

• Summarization is now generation

  dirkgr

  committed
  ce7f443

  Copy full SHA for ce7f443

Commits on Feb 25, 2022

• Support multiple answers in QA tasks

  dirkgr

  committed
  fc3216d

  Copy full SHA for fc3216d

• Compatibility with the latest Tango

  dirkgr

  committed
  114ebe3

  Copy full SHA for 114ebe3

• Update Tango version to a release version

  dirkgr

  committed
  ea5bfc8

  Copy full SHA for ea5bfc8

Commits on Feb 28, 2022

• Adds a perplexity task

  dirkgr

  committed
  ad93af9

  Copy full SHA for ad93af9

Commits on Mar 1, 2022

• More tasks

  dirkgr

  committed
  2329967

  Copy full SHA for 2329967

• Gitignore

  dirkgr

  committed
  8763df3

  Copy full SHA for 8763df3

• Beautification

  dirkgr

  committed
  4270ebf

  Copy full SHA for 4270ebf

• We updated Tango

  dirkgr

  committed
  20f9394

  Copy full SHA for 20f9394

• Don't call me Ludwig

  dirkgr

  committed
  58501c0

  Copy full SHA for 58501c0

• We're now called catwalk

  dirkgr

  committed
  08cafd5

  Copy full SHA for 08cafd5

• Tests!

  dirkgr

  committed
  e6046c5

  Copy full SHA for e6046c5

• Ignore vim

  dirkgr

  committed
  9b22a10

  Copy full SHA for 9b22a10

• Per-task metrics

  dirkgr

  committed
  ee7a0dd

  Copy full SHA for ee7a0dd

Commits on Mar 3, 2022

• Perplexity task

  dirkgr

  committed
  8aa38fd

  Copy full SHA for 8aa38fd