GPU-based FHE Example in LLM Use Case

[YY-OhioU]

Hi,

Thank you for sharing this example of integrating FHE with LLMs! I came across the use case examples (use_case_examples/llm) from this post: link.

The current implementation performs all model computations on the CPU. Would it be possible to provide an example demonstrating how to run the models on GPU?

Thanks again for your work!

[jfrery]

Hi @YY-OhioU,

You should be able to switch to GPU easily by adding use_gpu = True in the configuration in the compile call.

Note that you also need to install the gpu compatible concrete-python with

pip install concrete-python --extra-index-url https://pypi.zama.ai/gpu

(note that you might need to uninstall the version of concrete-python you have already installed).