needlebench， hf_deepseek_67b_base, needlebench_8k: 速度极慢，15个小时过去了，还只是评测了一半。这正常吗？正常的话有办法加速吗？

[ainilian]

command

CUDA_VISIBLE_DEVICES=4,5,6,7 python run.py configs/eval_needlebench.py --max-num-workers 4

gpu a100-40g

configs/eval_needlebench.py

from mmengine.config import read_base
with read_base():
    from opencompass.configs.models.deepseek.hf_deepseek_67b_base import models as hf_deepseek_67b_base

    # Evaluate needlebench_4k, adjust the configuration to use 8k, 32k, 128k, 200k, or 1000k if necessary.
    from opencompass.configs.datasets.needlebench.needlebench_8k.needlebench_8k import needlebench_datasets
    from opencompass.configs.summarizers.needlebench import needlebench_8k_summarizer as summarizer


datasets = sum([v for k, v in locals().items() if ('datasets' in k)], [])

for m in hf_deepseek_67b_base:
    m['path'] = '/base_models/deepseek-67b-base'
    m['max_seq_len'] = 16384
    m['max_out_len'] = 2000
    m['batch_size'] = 4

models = hf_deepseek_67b_base

work_dir = './outputs/needlebench'

[Mor-Li]

Hi @ainilian,

The slowness you’re experiencing could be related to the default inference setup. For faster evaluation, I recommend using lmdeploy or vllm, which are optimized for better performance on large models like hf_deepseek_67b_base.

You can refer to the following configurations to get started:

• lmdeploy configuration example: InternLM config
• vllm configuration example: InternLM config

Additionally, it seems like you are testing a base model (hf_deepseek_67b_base). Whie NeedleBench is designed to evaluate long-text capabilities, which usually require strong instruction-following abilities. For this reason, we generally recommend testing chat models instead of base models, as they are better suited for these kinds of evaluations.

Let me know if you need further assistance!