You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
Checklist
Describe the bug
使用模型:qwen2-14b-chat 量化版模型
在vllm和TurboMind的单请求耗时上,vllm耗时在5s左右,但turboMind推理耗时要接近10s
Reproduction
对应启动配置为:
backend_config = TurbomindEngineConfig(model_format='awq',
tp=1,
max_batch_size=1,
cache_max_entry_count=0.8)
pipe = pipeline(
model_path=self.model_path,
backend_config=backend_config
)
Environment
Error traceback
No response
The text was updated successfully, but these errors were encountered: