Skip to content

Adding Models

OnVeris runs AI models in the GGUF format, downloaded to your Node from Hugging Face — the main public library of AI models. You do this from the Gateway. Once a model is running on a Node, it will be available in OnVeris Chat, to Assistants and through the API.

What You Need

You will need two kinds of model:

  • A language model — for chat, Assistants and the API.
  • An embedding model — for Knowledge Bases and searching conversations. Every Node includes OnVeris Built-in Embeddings to start with; a dedicated embedding model gives better results.

Install A Backend

A backend is the software that runs models on your hardware. You must install one before you start a language model:

  1. On the Nodes page, click the edit icon on your Node.
  2. In the Executors panel, click the chip icon (Manage llama backends).
  3. Click Install beside the backend that suits your hardware. Vulkan is a safe default; choose CUDA for NVIDIA and ROCm for AMD. On a Mac, install Metal.

See Hardware and GPU for help choosing.

Download A Model

  1. In the Executors panel, click +.
  2. Search for a model — for example, Qwen3 — and click Search.
  3. Choose a result, then click Download beside a file. For most models, a file with Q4_K_M or Q5_K_M in its name is a good balance of quality and size.
  4. Wait for the download to finish. You can close the window — the download will carry on in the background.

The model will then appear in the Executors panel.

Note

Make sure the machine has enough memory for the model. As a rough guide, a 7–9 billion parameter model at Q4_K_M needs about 5 GB of GPU memory, plus room for its context window. See Hardware and GPU.

Start The Model

  1. Optionally, click the gear icon to check its settings. The Memory section shows whether it will fit.
  2. Click the play button.

The model will load, which can take a minute or more. When it shows Online, it is ready, and will appear in OnVeris Chat.

See Managing Models for everything else you can do with models.

If A Model Will Not Start

If a model fails to start, check the following:

  • Check that its backend is installed and Ready.
  • Check the memory estimate in its settings. If it says Does not fit, lower the context window, choose a smaller file, or store the KV cache at lower precision.
  • Look at the Node's Log for the error. See Diagnostics.

Adding Model Files By Hand

You can also copy .gguf files straight into the Node's models folder — for example, on a machine without internet access. See Model Discovery.