feat: 🎸 make the queue agnostic to the types of jobs

[severo]

Before we had two collections: for splits and for first-rows jobs. Now only one collection name "jobs", with a field "type". Note that the job arguments are still restricted to dataset (required) and optionally config and split.

BREAKING CHANGE: 🧨 two collections are removed and a new one is created. The function names have changed too.

[severo]

Note that we have to manually migrate the jobs from the splits queue and the first_rows queue to the new jobs queue.

To do so, in the prod mongo database:

• get the list of datasets with pending "splits" jobs

db.splits_jobs.distinct("dataset_name", {status: {"$in": ['started', 'waiting']}})

• get the list of datasets with pending "first_rows" jobs

db.first_rows_jobs.distinct("dataset_name", {status: {"$in": ['started', 'waiting']}})

Then relaunch them by sending a webhook:

DATASETS=(datasetA datasetB datasetC ...)
for dataset in ${DATASETS[@]}; do curl -X POST https://datasets-server.huggingface.co/webhook -H 'Content-Type: application/json' -d '{"event": "update", "repo": {"type": "dataset", "name": "'$dataset'"}}'; done;

Then: delete both collections

[severo]

Done with

DATASETS=(Graphcore/gqa-lxmert YWjimmy/PeRFception-v1 YWjimmy/PeRFception-v1-1 YWjimmy/PeRFception-v1-2 YWjimmy/PeRFception-v1-3 anonymousdeepcc/DeepCC echarlaix/gqa-lxmert huggingface/stable-diffusion-prompts)

and

DATASETS=(CodedotAI/code_clippy_github YWjimmy/PeRFception-v1 YWjimmy/PeRFception-v1-3 abdusahmbzuai/masc_dev common_voice icelab/ntrs_meta imthanhlv/binhvq_news21_raw mlsum munggok/KoPI-CC munggok/Oscar_Indo_May_2022 nishita/ade20k-sample openclimatefix/dgmr-mrms openclimatefix/gfs-surface-pressure-0.5deg openclimatefix/gfs-surface-pressure-1.0deg openclimatefix/gfs-surface-pressure-2.0deg openclimatefix/gfs-surface-pressure-2deg openclimatefix/nimrod-uk-1km openclimatefix/nimrod-uk-1km-test openclimatefix/nimrod-uk-1km-validation polinaeterna/earn raghav66/whisper-gpt tensorcat/wikipedia-japanese vialibre/spanish3bwc_extended_vocab)

[severo]

And I deleted the old collections