Workaround: Build failure with CUDA 13.0: "Unsupported CUDA version: 13" error in onnxruntime-node

[erkkimon]

Environment:

• OS: Arch Linux
• CUDA: 13.0.88 (nvcc release 13.0, V13.0.88)
• Node.js: v20.19.5
• Package: @huggingface/transformers → onnxruntime-node

Problem Description:

When installing dependencies on a system with CUDA 13, npm install fails during the onnxruntime-node post-install script with the following error:

npm error Failed to detect CUDA version from `nvcc --version`: Unsupported CUDA version: 13
npm error node:events:502
npm error       throw er; // Unhandled 'error' event
npm error       ^
...
npm error Error: Failed to find specific package versions for Microsoft.ML.OnnxRuntime.Gpu.Linux in https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/nuget/v3/index.json 
npm error     at nugetPackageUrlResolver (/home/lab/Git/Perplexica/node_modules/onnxruntime-node/script/install-utils.js:129:13)

The error Failed to detect CUDA version from nvcc --version: Unsupported CUDA version: 13 occurs because onnxruntime-node doesn't yet have pre-built binaries for CUDA 13.

Workaround - Force CPU Mode Installation:

1. Clean previous installation attempts:

rm -rf node_modules package-lock.json

2. Install dependencies while skipping post-install scripts (this prevents onnxruntime-node from attempting to download unsupported GPU binaries):

npm install --ignore-scripts

3. Rebuild the better-sqlite3 native bindings manually (required for database operations):

npm rebuild better-sqlite3

4. Build and start the application normally:

npm run build
npm run start

Important Limitation:
This workaround installs onnxruntime-node in CPU-only mode so any local model inference using @huggingface/transformers will run on CPU, which may be slower for embeddings and other operations.

Alternative Solution for GPU Acceleration:

If you need GPU acceleration for embeddings but cannot downgrade CUDA 13, I recommend using Ollama for embeddings while running vllama for LLM inference which is is a drop-in replacement for Ollama that runs inference on GPU while maintaining full API compatibility.

My Working Configuration:

I'm successfully using this setup with Perplexica with no speed penalty:

• Embeddings: ollama running nomic-embed-text:latest on CPU
• LLM Inference: vllama running huihui_ai/deepseek-r1-abliterated:14b on GPU

This separation provides optimal performance:

• Embeddings are fast enough on CPU with Ollama
• LLM inference gets full GPU acceleration via vllama
• No model swapping delays since each service runs its own dedicated model

To configure this in Perplexica:

1. Install both Ollama and vllama
2. Pull model for embeddings using ollama: ollama pull nomic-embed-text:latest
3. Pull model for LLM using ollama: ollama pull huihui_ai/deepseek-r1-abliterated:14b
4. In Perplexica settings:
   • Set Embedder Provider to "Ollama" (uses CPU for embeddings) using port http://localhost:11434
   • Set LLM Provider to "LiteLLM" but point it to vllama's API endpoint http://localhost:11435
5. Both services can run simultaneously on different ports

I hope Perplexica will work soon with Cuda 13, but at least here is a workaround until that!