[Model] Enable optional prefix when loading embedding models #10639

DarkLight1337 · 2024-11-25T15:55:20Z

Some embedding models use the checkpoint of *ForCausalLM, while others use *Model, yet their architecture names might not always match the expected weights. To improve flexibility, this PR enables loading embedding models (*EmbeddingModel in vLLM) using the weights of either checkpoint format.

FIX #10193 (comment)

Signed-off-by: DarkLight1337 <[email protected]>

github-actions · 2024-11-25T15:55:35Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

vllm/model_executor/models/roberta.py

Isotr0py

LGTM!

ra-MANUJ-an · 2024-11-26T13:32:19Z

Is there any code example to follow to get embeddings?

DarkLight1337 · 2024-11-26T13:36:58Z

See https://github.com/vllm-project/vllm/blob/main/examples%2Foffline_inference_embedding.py

ra-MANUJ-an · 2024-11-26T13:38:55Z

@DarkLight1337 It gives me following error:

RuntimeError: stack expects each tensor to be equal size, but got [6, 32000] at entry 0 and [8, 32000] at entry 1

DarkLight1337 · 2024-11-26T13:40:32Z

@DarkLight1337 It gives me following error:

RuntimeError: stack expects each tensor to be equal size, but got [6, 32000] at entry 0 and [8, 32000] at entry 1

Can you open a new issue and provide more details there?

ra-MANUJ-an · 2024-11-26T13:49:22Z

I opened a new issue #10673 Please check.

…oject#10639) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

githebs · 2024-12-10T08:59:09Z

Hi @DarkLight1337
Thanks for the effort, any idea when we will see a new build with this fix included, right now i have no idea how to test in docker while waiting (maybe source compile with dockerfile.cpu ?)

DarkLight1337 · 2024-12-10T09:04:46Z

You can try the Dockerfile in this section

githebs · 2024-12-10T10:07:00Z

docker compose

services:
  vllm:
      container_name: vllm-cpu-embed-kalm
      image: public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:6faec545057e6152e92e8ab619fc018e20864943
      restart: unless-stopped
      shm_size: '64gb'
      command: ["vllm", "serve", "/models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1", "--max-model-len", "32768", "--device", "cpu", "--task", "embedding"]

error output

[+] Running 1/1
 ✔ Container vllm-cpu-embed-kalm  Started                                                                                                           5.3s 
docker logs -f vllm-cpu-embed-kalm
INFO 12-10 01:56:20 api_server.py:625] vLLM API server version 0.6.4.post2.dev286+g6faec545
INFO 12-10 01:56:20 api_server.py:626] args: Namespace(subparser='serve', model_tag='/models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1', config='', host=None, port=8000, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1', task='embedding', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=32768, guided_decoding_backend='xgrammar', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='cpu', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x7f2ec42b7f60>)
INFO 12-10 01:56:20 api_server.py:197] Started engine process with PID 16
INFO 12-10 01:56:20 config.py:1844] Downcasting torch.float32 to torch.float16.
INFO 12-10 01:56:23 config.py:1844] Downcasting torch.float32 to torch.float16.
Traceback (most recent call last):
  File "/usr/local/bin/vllm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/scripts.py", line 201, in main
    args.dispatch_function(args)
  File "/usr/local/lib/python3.12/dist-packages/vllm/scripts.py", line 42, in serve
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 649, in run_server
    async with build_async_engine_client(args) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 116, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 200, in build_async_engine_client_from_engine_args
    engine_config = engine_args.create_engine_config()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1202, in create_engine_config
    config = VllmConfig(
             ^^^^^^^^^^^
  File "<string>", line 18, in __init__
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 2484, in __post_init__
    self.model_config.verify_async_output_proc(self.parallel_config,
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 516, in verify_async_output_proc
    if not current_platform.is_async_output_supported(self.enforce_eager):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/interface.py", line 159, in is_async_output_supported
    raise NotImplementedError
NotImplementedError
Process SpawnProcess-1:
ERROR 12-10 01:56:27 engine.py:366] 
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
    engine_config = engine_args.create_engine_config(usage_context)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1202, in create_engine_config
    config = VllmConfig(
             ^^^^^^^^^^^
  File "<string>", line 18, in __init__
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 2484, in __post_init__
    self.model_config.verify_async_output_proc(self.parallel_config,
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 516, in verify_async_output_proc
    if not current_platform.is_async_output_supported(self.enforce_eager):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/interface.py", line 159, in is_async_output_supported
    raise NotImplementedError
NotImplementedError
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 368, in run_mp_engine
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 357, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 114, in from_engine_args
    engine_config = engine_args.create_engine_config(usage_context)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1202, in create_engine_config
    config = VllmConfig(
             ^^^^^^^^^^^
  File "<string>", line 18, in __init__
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 2484, in __post_init__
    self.model_config.verify_async_output_proc(self.parallel_config,
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 516, in verify_async_output_proc
    if not current_platform.is_async_output_supported(self.enforce_eager):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/platforms/interface.py", line 159, in is_async_output_supported
    raise NotImplementedError
NotImplementedError

I had to use the latest ✅ commit from main since yours (fe25236) was not able to download

Also i had to use commands with vllm serve instead of the command: --host 0.0.0.0 --model /models/HIT-TMG-KaLM-embedding-multilingual-mini-instruct-v1 --max-model-len 32768 --device cpu --task embedding for some reason ?

To my knowledge the embedding model i use (https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1) is trained from Qwen/Qwen2-0.5B

DarkLight1337 · 2024-12-10T10:15:40Z

Can you go into the Docker container and show the output of collect_env.py? Seems that vLLM can't detect your hardware.

githebs · 2024-12-10T10:35:45Z

Output

Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.12.8 (main, Dec  4 2024, 08:54:12) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-4.18.0-553.16.1.el8_10.x86_64-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        45 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               4
On-line CPU(s) list:                  0-3
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   1
Core(s) per socket:                   1
Socket(s):                            4
Stepping:                             7
BogoMIPS:                             5985.93
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Hypervisor vendor:                    VMware
Virtualization type:                  full
L1d cache:                            192 KiB (4 instances)
L1i cache:                            128 KiB (4 instances)
L2 cache:                             5 MiB (4 instances)
L3 cache:                             156 MiB (4 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-3
Vulnerability Gather data sampling:   Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:          KVM: Mitigation: VMX unsupported
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] mypy==1.11.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] sentence-transformers==3.2.1
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.46.3
[pip3] transformers-stream-generator==0.0.5
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post2.dev286+g6faec545
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect

NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
NVIDIA_DRIVER_CAPABILITIES=compute,utility
CUDA_VERSION=12.4.1
NVIDIA_DISABLE_REQUIRE=1
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64

I did not add a gpu on purpose since the goal is cpu embedding serving.

DarkLight1337 · 2024-12-10T10:59:39Z

@Isotr0py can you help look into this?

Isotr0py · 2024-12-10T11:17:14Z

@githebs Seems that the docker image is not cpu installation, you need to use a cpu installation image built from Dockerfile.cpu: https://docs.vllm.ai/en/latest/getting_started/cpu-installation.html#quick-start-using-dockerfile

…oject#10639) Signed-off-by: DarkLight1337 <[email protected]>

githebs · 2024-12-16T08:37:16Z

@githebs Seems that the docker image is not cpu installation, you need to use a cpu installation image built from Dockerfile.cpu: https://docs.vllm.ai/en/latest/getting_started/cpu-installation.html#quick-start-using-dockerfile

I have no idea where to get that for the latest releases since the doc states "vLLM provides wheelsfor Linux running on a x86 platform with CUDA 12" so CPU is out of the equation there. (unless full build)

Is there a timeline for a new release with that fix ? And if so is this embedding model, derived from Qwen, supported ?
https://huggingface.co/HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1

Thanks

Isotr0py · 2024-12-16T08:59:16Z

Hmmm, we only release pre-built wheel for GPU currently...

So if you want to serve models with CPU, you need to use CPU docker image (this is always synced with latest commit) or build CPU backend from source manually (this should be fast within ~5min).

…oject#10639) Signed-off-by: DarkLight1337 <[email protected]>

Enable optional prefix

fe25236

Signed-off-by: DarkLight1337 <[email protected]>

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 25, 2024

DarkLight1337 requested a review from Isotr0py November 25, 2024 15:55

DarkLight1337 commented Nov 25, 2024

View reviewed changes

vllm/model_executor/models/roberta.py Show resolved Hide resolved

DarkLight1337 mentioned this pull request Nov 25, 2024

[Hardware][CPU] Add embedding models support for CPU backend #10193

Merged

Isotr0py approved these changes Nov 25, 2024

View reviewed changes

Isotr0py enabled auto-merge (squash) November 25, 2024 16:10

Isotr0py merged commit cf73f0c into vllm-project:main Nov 25, 2024
54 checks passed

DarkLight1337 mentioned this pull request Nov 26, 2024

[Feature]: Add embeddings api for Llama #6947

Closed

DarkLight1337 deleted the robust-embedding-weights branch November 26, 2024 02:21

afeldman-nm pushed a commit to neuralmagic/vllm that referenced this pull request Dec 2, 2024

[Model] Enable optional prefix when loading embedding models (vllm-pr…

80a1dd4

…oject#10639) Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Andrew Feldman <[email protected]>

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024

[Model] Enable optional prefix when loading embedding models (vllm-pr…

0d59963

…oject#10639) Signed-off-by: DarkLight1337 <[email protected]>

anko-intel pushed a commit to HabanaAI/vllm-fork that referenced this pull request Feb 12, 2025

[Model] Enable optional prefix when loading embedding models (vllm-pr…

aa568c2

…oject#10639) Signed-off-by: DarkLight1337 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Model] Enable optional prefix when loading embedding models #10639

[Model] Enable optional prefix when loading embedding models #10639

DarkLight1337 commented Nov 25, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 25, 2024

Isotr0py left a comment

ra-MANUJ-an commented Nov 26, 2024

DarkLight1337 commented Nov 26, 2024

ra-MANUJ-an commented Nov 26, 2024

DarkLight1337 commented Nov 26, 2024

ra-MANUJ-an commented Nov 26, 2024

githebs commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

githebs commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

githebs commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

Isotr0py commented Dec 10, 2024

githebs commented Dec 16, 2024

Isotr0py commented Dec 16, 2024

[Model] Enable optional prefix when loading embedding models #10639

[Model] Enable optional prefix when loading embedding models #10639

Conversation

DarkLight1337 commented Nov 25, 2024 • edited by github-actions bot Loading

github-actions bot commented Nov 25, 2024

Isotr0py left a comment

Choose a reason for hiding this comment

ra-MANUJ-an commented Nov 26, 2024

DarkLight1337 commented Nov 26, 2024

ra-MANUJ-an commented Nov 26, 2024

DarkLight1337 commented Nov 26, 2024

ra-MANUJ-an commented Nov 26, 2024

githebs commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

githebs commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

githebs commented Dec 10, 2024

DarkLight1337 commented Dec 10, 2024

Isotr0py commented Dec 10, 2024

githebs commented Dec 16, 2024

Isotr0py commented Dec 16, 2024

DarkLight1337 commented Nov 25, 2024 •

edited by github-actions bot

Loading