Automatic update tasks list

[Samoed]

How about automating the update of the tasks list from the mteb library? A separate script could be created to run before refresh.py to handle this task.

@orionw

[orionw]

Hi @Samoed, would love to make it more automatic. For most of the leaderboard though the tasks are pre-defined from some set of results (e.g. English MTEB is pre-defined, French is pre-defined, etc.). I think the results objects that gets pulled in has all the tasks that are run, but then we filter them in the refresh.py script to the benchmarks we care about.

We do want to transition to a "create-your-own" style leaderboard where you can select the tasks though. And for that automatically updating the tasks seems very important!

Perhaps I'm misunderstanding - let me know if this is not what you were meaning?

[Samoed]

My intention was to automatically include tasks for each language from mteb. However, if you're currently using only a subset of those tasks, then my suggestion may not be applicable for this repository, but for suitable for create-your-own.