[Bug] Failure to Dispatch Head Dimension 80 in sglang with Specific Configurations #1109
Closed
5 tasks done
Comments
|
By the way, I'm waiting to fix the failure to Dispatch head_dim=80, and then I'll merge the model support local branch into the online main branch. |
|
Consider trying support in FlashInfer or the Triton kernel. From the perspective of rapid verification, I suggest you modify the Triton kernel. |
3 tasks
I hit a different error. not sure what is wrong on my side. |
|
I am fixing the kernel for non 2^n case. Happy to resolve this together, but i am not able to reproduce. |
|
@ByronHsu you can try with the existing stablelm model code (which already supports stablelm-2) and the stabilityai/stablelm-3b-4e1t model name. The latter also has a head dim of 80. Your fix makes it work. |
3 tasks
|
@zhyncs we can close this now |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
Describe the bug
Issue Description:
When running sglang with hidden_dim set to 80, the following exceptions are encountered under different configurations:
With enable_cuda_graph set to True:
The complete stack information is as follows:
With disable_cuda_graph set to True:
The complete stack information is as follows:
With disable_flashinfer set to True:
AssertionError: assert Lq in {16, 32, 64, 128, 256, 576}, where Lq = 80.The complete stack information is as follows:
Reproduction
run command
model
xverse/XVERSE-MoE-A4.2B-Chat
Steps to Reproduce:
I am currently supporting XVERSE models in a local branch and encountered an issue where the server failed to launch. Although the issue cannot be replicated directly, possible solutions may be inferred from the error message and the configuration file.
Below is the config.json file, where the
hidden_size/num_attention_heads = 2560/32= 80.{ "architectures": [ "XverseMoEForCausalLM" ], "auto_map": { "AutoConfig": "configuration_xverse.XverseConfig", "AutoModelForCausalLM": "modeling_xverse.XverseForCausalLM" }, "pad_token_id": 1, "bos_token_id": 2, "eos_token_id": 3, "hidden_act": "silu", "hidden_size": 2560, "initializer_range": 0.02, "intermediate_size": 1728, "max_position_embeddings": 8192, "max_tokenizer_truncation": 6144, "model_type": "xverse", "num_attention_heads": 32, "num_hidden_layers": 28, "rms_norm_eps": 1e-06, "tie_word_embeddings": false, "rope_theta": 500000, "moe_top_k": 6, "num_experts": 64, "num_shared_experts": 2, "output_router_logits": false, "router_aux_loss_coef": 0.01, "torch_dtype": "bfloat16", "transformers_version": "4.38.2", "use_cache": true, "vocab_size": 100534, "_attn_implementation": "eager" }Environment
Python: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0] CUDA available: True GPU 0: NVIDIA A800-SXM4-80GB GPU 0 Compute Capability: 8.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 470.182.03 PyTorch: 2.3.1+cu121 sglang: 0.2.12 flashinfer: 0.1.5+cu121torch2.3 triton: 2.3.1 transformers: 4.43.3 requests: 2.32.3 tqdm: 4.66.4 numpy: 1.26.4 aiohttp: 3.9.5 fastapi: 0.111.1 hf_transfer: 0.1.8 huggingface_hub: 0.24.2 interegular: 0.3.3 packaging: 24.1 PIL: 10.4.0 psutil: 6.0.0 pydantic: 2.8.2 uvicorn: 0.30.3 uvloop: 0.19.0 zmq: 26.0.3 vllm: 0.5.3.post1 multipart: 0.0.9 openai: 1.37.1 anthropic: 0.31.2 NVIDIA Topology: GPU0 CPU Affinity NUMA Affinity GPU0 X 116-231 1 Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks ulimit soft: 1048576The text was updated successfully, but these errors were encountered: