[Offline model] admin setting

[andreiramani]

Is there any documentation about how to using offline model (safetensor/gguf) in khoj?

I tried to use http://localhost:42110/server/admin/database/chatmodel/add/ and successfully download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF.

Unfortunately, when using chat - there is no response, with python error in khoj cmd console ValueError: Failed to create llama_context

Note:

• self-host python package (localhost with anaconda)
• windows 11 x64
• cuda 12.4