Can I set the embedding model in CPU memory and the LLM model in GPU with Ollama?

[ShiShuyang]

Limited by the GPU memory, I have to move the embedding model to CPU memory. How can I deploy it with Ollama? Many thanks for your help!

[ZhenhangTung]

pls check https://ragflow.io/docs/dev/deploy_local_llm @ShiShuyang

[ShiShuyang]

Thank you for your reply! But this document seems that there is no information about how to set one embedding model in CPU and another LLM model in GPU.

[ZhenhangTung]

Got it. I believe the key would be achieving this goal using Ollama? Pls check if https://github.com/copilot/share/884e1314-03c0-8ce1-b143-3a4360546806 helps and do some search around this topic?

[ShiShuyang]

Under the help of ChatGPT, this problem has been solved.
First, create a plain text file (for example, named ollama_embedding_cpu.txt) with the following content:

FROM modelscope.cn/Qwen/Qwen3-Embedding-8B-GGUF:Qwen3-Embedding-8B-f16.gguf
PARAMETER num_gpu 0

Then, run the following command to create the model in Ollama:

ollama create qwen3-embed-cpu -f ollama_embedding_cpu.txt

After that, you can run the model using:

ollama run qwen3-embed-cpu

This will force the model to run on CPU only, which is useful if you don't have a GPU or want to test performance in a CPU-only environment.