Error while evaluating SouthAfricanLangClassification: 'default'

[sigridjineth]

Error while evaluating SouthAfricanLangClassification: 'default'
Error during execution: 'default'
Traceback (most recent call last):
  File "/workspace/sigrid/mteb-hacking/evaluation_3.py", line 190, in <module>
    results = evaluation.run(model, output_folder="results/Ensemble-stella_gemma2_all_en")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/sigrid/mteb-hacking/.venv/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 570, in run
    raise e
  File "/workspace/sigrid/mteb-hacking/.venv/lib/python3.11/site-packages/mteb/evaluation/MTEB.py", line 548, in run
    new_results = TaskResult.from_task_results(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/sigrid/mteb-hacking/.venv/lib/python3.11/site-packages/mteb/load_results/task_results.py", line 177, in from_task_results
    eval_langs = subset2langscripts[hf_subset]
                 ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'default'

I think there is some key error happens in the dataset needed to be evaluated.

The script that I ran is for English tasks.

    try:
        print("Initializing EnsembleModel...")
        model = Model(..., ...)

        print("Setting up MTEB evaluation...")
        evaluation = MTEB(task_langs=['en'])

        print("Running evaluation...")
        results = evaluation.run(model, output_folder="results/...")
        print("Results:", results)