Trying to understand JAN AI Server deployment option to offload GPU to external machine

[DPBattaglia]

Hi team,

I’m very interested in trying out Jan Server for our company as a replacement for OpenWebUI (OWUI). We currently have OWUI operational, but as a small business (fewer than 5 employees), we’ve found it too complex to fine-tune and maintain. Jan AI’s simplicity — and the major improvements over the past few months — make it look like a fantastic alternative for SMBs like ours.

After reviewing the guides and Docker Compose examples, I’m trying to determine what would be the most optimal deployment architecture for our setup.

🧩 Overview

• Business type: Small business (<5 employees)
• Hardware: Mac M3 Ultra (used as inference machine)
• Infrastructure: Synology NAS (handles reverse proxy, MS365 EntraID SSO, file storage, and local data access)
  Current setup:
• Synology manages reverse proxy + SSO
• Mac runs an exposed OpenAI-compatible MLX server to handle AI inference requests

We’d like to replicate this setup with Jan Server, ideally with a simpler, more maintainable configuration.

⚙️ Deployment Questions

I’ve reviewed your docker directory and would appreciate some guidance on how best to structure our setup:

1. Should I use jan-server/docker/infrastructure.yml as the main template?

• I’m unsure whether this file should also include the llm-api service.

2. On the Mac M3 Ultra, should I run one or both of:

• llm-api (/service-api.yml), and/or
• vllm (/inference.yml)

3. How should I connect all the components together — via the .env file?
4. Do I still need Kong if the Synology NAS is already acting as our reverse proxy (terminating HTTPS connections and forwarding requests)?

• It looks like Kong might still be required to handle API routing between services.

5. Would it make more sense to run llm-api on the Synology NAS or on the Mac M3?
6. Is the vllm inference setup on macOS tuned for Metal to utilize the M3 Ultra’s GPU and CPU efficiently?

• Or should I continue using an MLX-based OpenAI-compatible server instead?

7. Do I need Keycloak for OIDC / SSO integration with Microsoft EntraID, or can Jan Server handle that natively?
8. Side question — will Jan eventually expose the OpenAI v1/audio/transcriptions API (ASR)?

I’d love to consolidate everything into Jan AI for our SMB AI platform.

🖥️ Desired Architecture

Ideally, we’d like:

• Synology NAS → runs Jan front-end, SSO integration, DB, chat history, and file management
• Mac M3 Ultra → handles all GPU-intensive inference tasks

Based on what I’ve read, my assumption is that:

• Synology would run a combined infrastructure.yml + service-api.yml setup (without inference),
• and the Mac would run inference.yml for model management and inference hosting.

Does that sound correct?

Also, would the Jan web GUI (and user sign-in) still be accessible via the default web interface on port 8000?

🙏 Closing

Thanks for building such an awesome product — it’s clearly come a long way recently.
I really appreciate your time in helping me confirm the best approach for an SMB setup like ours.

Kind regards,
David

[locnguyen1986]

@DPBattaglia ,Thanks for your interest in Jan-Server!

Before answering your questions, just one note: Jan-Server is still being set up. We’re planning to add a lightweight mode soon, which should better fit your needs.

Currently, Jan-Server includes rich features such as self-hosted /mcp, self-hosted /responses and /media services, these may be heavier than what you need for a lightweight chat using vLLM.
The lightweight setup (an LLM-API service with a simple Chat Web UI) is on our roadmap, and the current Chat UI is migrating into this project.

For your questions

1. Should I use jan-server/docker/infrastructure.yml as the main template?

Yes. Use infrastructure.yml for PostgreSQL, Keycloak, and Kong.
Then merge it with services-api.yml to include llm-api, media-api, and response-api.
Do not include inference.yml on Synology.

2. Should infrastructure.yml also include the llm-api service?

No. infrastructure.yml is only for infrastructure (DB, auth, gateway).
Add llm-api from services-api.yml as a separate service when you combine files.

3. On the Mac M3 Ultra, should I run llm-api and/or vllm?

Neither. Keep your existing MLX server running natively on the Mac (outside Docker).
The vLLM setup in inference.yml is CUDA-only and won’t use Metal GPU.
MLX is already optimized for the M3 Ultra.

4. How should I connect all components, via the .env file?

Yes. Use .env to configure:

• JAN_DEFAULT_NODE_URL=http://mac-m3-ultra.local:8000 → points to your MLX server
• Database credentials
• Kong/Keycloak URLs
• Auth settings

5. Do I still need Kong if Synology is already the reverse proxy?

Yes.
Synology terminates HTTPS and forwards traffic to Kong, which handles internal API routing and JWT validation.
Flow:
Synology (HTTPS) → Kong :8000 → Internal Services

6. Should llm-api run on Synology or Mac M3?

Synology.
llm-api is just an orchestration layer (no GPU required).
Your Mac M3 runs inference only.
llm-api on Synology forwards inference requests to the MLX server on your Mac.

7. Is vllm inference tuned for Metal on the M3 Ultra?

No.
vLLM supports CUDA only (NVIDIA GPUs), it doesn’t use Metal acceleration.
Stick with MLX, which is Metal-optimized.

8. Do I need Keycloak for OIDC/SSO with Microsoft EntraID?

Recommended but optional.

• With Keycloak: Federate to EntraID for easier token management and guest access.
• Without Keycloak: Configure Kong to validate EntraID JWTs directly (set JWKS_URL to EntraID endpoint).
  Lighter setup, but tradeoff that no guest/API key management.

9. Will Jan expose OpenAI /v1/audio/transcriptions (ASR)?

Not yet.
Jan-Server doesn’t currently support /v1/audio/transcriptions.

---

Desired Architecture Confirmation

Your assumption:

Synology: infrastructure.yml + service-api.yml (no inference)
Mac: inference.yml for model management

Correct, with one tweak:

• Synology: infrastructure.yml + services-api.yml + services-mcp.yml
  (full stack except inference)
• Mac: Run native MLX server, skip inference.yml entirely.

[DPBattaglia]

@locnguyen1986 - Amazing thanks so much for the understanding - much clearer and great to know we can essentially run the full Jan stack on the synology and keep the inference pathway the same.

Let me know when you get the lightweight version up and running to - that would be great to play around with.