Inference Llama3-70b has an AssertionError

[Ikkyu321]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

I run sglang.server, but get an assertion error.

Reproduction

python3 -m sglang.launch_server --model-path ./models/Meta-Llama-3-70B-Instruct --host 0.0.0.0 --port 30000 --tp 8 --mem-fraction-static 0.7 --chunked-prefill-size 8192

Environment

Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A40
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.3, V12.3.107
CUDA Driver Version: 535.161.08
535.161.08
535.161.08
535.161.08
535.161.08
535.161.08
535.161.08
535.161.08
PyTorch: 2.3.1+cu121
sglang: 0.2.7
flashinfer: 0.1.3+cu121torch2.3
requests: 2.32.3
tqdm: 4.66.4
numpy: 1.24.4
aiohttp: 3.9.1
fastapi: 0.110.1
hf_transfer: 0.1.8
huggingface_hub: 0.24.5
interegular: 0.3.3
packaging: 23.2
PIL: 9.5.0
psutil: 5.9.4
pydantic: 2.7.0
uvicorn: 0.29.0
uvloop: 0.19.0
zmq: 25.1.2
vllm: 0.5.3.post1
openai: 1.37.1
anthropic: 0.32.0
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     PHB     PHB     PHB     PHB     PHB     PHB     0-85    0               N/A
GPU1    PHB      X      PHB     PHB     PHB     PHB     PHB     PHB     0-85    0               N/A
GPU2    PHB     PHB      X      PHB     PHB     PHB     PHB     PHB     0-85    0               N/A
GPU3    PHB     PHB     PHB      X      PHB     PHB     PHB     PHB     0-85    0               N/A
GPU4    PHB     PHB     PHB     PHB      X      PHB     PHB     PHB     0-85    0               N/A
GPU5    PHB     PHB     PHB     PHB     PHB      X      PHB     PHB     0-85    0               N/A
GPU6    PHB     PHB     PHB     PHB     PHB     PHB      X      PHB     0-85    0               N/A
GPU7    PHB     PHB     PHB     PHB     PHB     PHB     PHB      X      0-85    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

ulimit soft: 1048576

[Ying1123]

Thanks for reporting this. However, our initial code does not print enough error messages for warmup errors.
Could you apply this change and see what the error messages are?

#932