Import EAI tasks

[ibeltagy]

EAI tasks that are not on CrossFit.
Total task files in EAI: 47
Missing from CrossFit: 28

Not on HF dataset

• arithmetic
• asdiv - A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
• coqa
• gsm8k - Training Verifiers to Solve Math Word Problems
• hendrycks_ethics, math, test
• lambada-cloze
• lambada-multilingual
• logiqa
• mutual - MuTual: A Dataset for Multi-Turn Dialogue Reasoning
• naturalqs
• pile
• qa4mre
• quac
• sat
• storycloze
• translation
• triviaqa
• truthfulqa
• unscramble
• wikitext

Available on HF dataset

• head_qa
• lambada
• prost - PROST: Physical Reasoning about Objects Through Space and Time
• pubmedqa
• qasper
• wsc273 - another version of winogrande
• cbt - Children’s Book
• drop

[ibeltagy]

Not all the tasks listed above are important. Listing the important ones that are not on CrossFit and that we still need to import from EAI.

• arithmetic
• coqa
• drop
• logiqa
• naturalqs
• sat
• storycloze
• triviaqa
• wikitext
• head_qa
• lambada
• prost
• pubmedqa
• qasper

[ibeltagy]

Also, looking at the prompts of EAI, many of them are either no prompt at all (arithmetic, wikitext, lambada) or has a mostly unified QA format

[dirkgr]

Superseded by #8.