Adding Models
OnVeris runs AI models in the GGUF format, downloaded to your Node from Hugging Face — the main public library of AI models. You do this from the Gateway. Once a model is running on a Node, it will be available in OnVeris Chat, to Assistants and through the API.
What You Need
You will need two kinds of model:
- A language model — for chat, Assistants and the API.
- An embedding model — for Knowledge Bases and searching conversations. Every Node includes OnVeris Built-in Embeddings to start with; a dedicated embedding model gives better results.
Install A Backend
A backend is the software that runs models on your hardware. You must install one before you start a language model:
- On the Nodes page, click the edit icon on your Node.
- In the Executors panel, click the chip icon (Manage llama backends).
- Click Install beside the backend that suits your hardware. Vulkan is a safe default; choose CUDA for NVIDIA and ROCm for AMD. On a Mac, install Metal.
See Hardware and GPU for help choosing.
Download A Model
- In the Executors panel, click +.
- Search for a model — for example,
Qwen3— and click Search. - Choose a result, then click Download beside a file. For most models, a file with Q4_K_M or Q5_K_M in its name is a good balance of quality and size.
- Wait for the download to finish. You can close the window — the download will carry on in the background.
The model will then appear in the Executors panel.
Note
Make sure the machine has enough memory for the model. As a rough guide, a 7–9 billion parameter model at Q4_K_M needs about 5 GB of GPU memory, plus room for its context window. See Hardware and GPU.
Start The Model
- Optionally, click the gear icon to check its settings. The Memory section shows whether it will fit.
- Click the play button.
The model will load, which can take a minute or more. When it shows Online, it is ready, and will appear in OnVeris Chat.
See Managing Models for everything else you can do with models.
If A Model Will Not Start
If a model fails to start, check the following:
- Check that its backend is installed and Ready.
- Check the memory estimate in its settings. If it says Does not fit, lower the context window, choose a smaller file, or store the KV cache at lower precision.
- Look at the Node's Log for the error. See Diagnostics.
Adding Model Files By Hand
You can also copy .gguf files straight into the Node's models folder — for example, on a machine without internet access. See Model Discovery.