Skip to content

Chat Completions

The chat completions endpoint generates a response from a model or an Assistant. It follows the OpenAI Chat Completions API, so you can use it with the OpenAI libraries and with any framework that supports them.

Endpoint

POST /v1/chat/completions

The endpoint requires the chat.completions scope. See Authentication.

Request Body

Field Type Required Description
model string Yes What should answer: a model ID, a model name, or an Assistant such as assistant:hr-helper. See Listing Models
messages array Yes The conversation. Each item has a role (system, user, assistant or tool) and content
stream boolean No If true, the response is streamed as Server-Sent Events. Default false
temperature number No Randomness, 0–2. Higher is more varied
top_p number No Nucleus sampling threshold
max_completion_tokens integer No The most tokens the response may contain. max_tokens is accepted too
stop string or array No Sequences where generation stops
reasoning_effort string No How much a reasoning model may think first: minimal, low, medium or high. See Reasoning
response_format object No {"type": "json_object"} or a json_schema for structured output
tools array No Tool definitions the model may call. See Tools
tool_choice string or object No "auto", "none", "required", or a specific tool
frequency_penalty number No Reduces repetition of frequent tokens, -2 to 2
presence_penalty number No Reduces repetition of any earlier token, -2 to 2
seed integer No Makes sampling repeatable

Settings you leave out will use the model's own defaults, set by your administrator.

Basic Example

Request:

{
  "model": "fp:9189ab12cd34ef56",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What is the capital of France?" }
  ]
}

Response:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1718000000,
  "model": "Qwen3.5-9B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "completion_tokens": 9,
    "total_tokens": 37
  }
}

Streaming

You can set "stream": true to receive the response as Server-Sent Events. Each event will carry a part of the response.

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant","content":"Why"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" don't"},"index":0}]}

...

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{},"finish_reason":"stop","index":0}]}

data: [DONE]

Note

With response_format set to json_object, the response will not be streamed, even if stream is true.

Using An Assistant

You can send a request to an Assistant instead of a model. Set model to an Assistant's reference, such as assistant:hr-helper. The Assistant will answer with its own model, instructions, tools and Knowledge Bases. The Assistant must be available on the API, and shared with your key or user.

For example:

{
  "model": "assistant:hr-helper",
  "messages": [{ "role": "user", "content": "How many days of annual leave do I get?" }]
}
  • The Assistant's temperature and reasoning effort, if set, take precedence over the request's.
  • What happens to a system message you send depends on the Assistant's settings: it may be added to the Assistant's instructions, ignored, used instead, or refused. See Assistants.
Error Status Meaning
model_not_found 404 No such Assistant or model — or it is not available to you, or not on the API
model_not_available 503 The Assistant exists, but none of its models are running
system_prompt_not_allowed 400 The Assistant refuses requests that bring their own system prompt

Images

Models that can read images accept them as message content parts. The image must be inline, as a data: URI. For example:

{
  "model": "fp:9189ab12cd34ef56",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What does this error message say?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KGgo..." } }
    ]
  }]
}

Check a model's input_modalities in Listing Models to see whether it reads images. Requests will be refused with 400 and the code unsupported_media when:

  • an image is given as a web link — send it inline instead
  • the model is known not to read images

Reasoning

For models that think before answering, you can use reasoning_effort to set how much they think:

Value Effect
(not sent) The model's own setting
minimal No thinking — answers straight away
low Thinking capped at about 1,000 tokens
medium Thinking capped at about 4,000 tokens
high Thinks as long as it needs

A capped model answers sooner, but hard questions are more likely to come out wrong.

Tools

A request can use two kinds of tool: tools the Gateway runs for you, and your own tools, which you run yourself.

Tools The Gateway Runs

Your administrator may make tools available on the API — Connectors, MCP servers, Knowledge Base search, or the tools of an Assistant you call. When there are any, OnVeris runs the tool loop itself: the model calls a tool, the Gateway runs it, gives the model the result, and repeats until the model answers — or calls one of your own tools. You will receive only the final answer.

When streaming, the response will then also carry named events showing the tool activity, alongside the usual data: chunks. For example:

event: agent_tool_start
data: {"call_id":"call_1","tool_name":"search_knowledge_base","args":"{\"query\":\"annual leave\"}"}

event: agent_tool_result
data: {"call_id":"call_1","result":"…","duration_ms":412,"error":null}
Event When
agent_thinking_start A round of the loop begins
agent_tool_start A tool is called
agent_tool_result A tool returns
agent_note Something the caller should know — for example earlier tool results were dropped to fit the context window, or the loop hit its limit
agent_loop_end The loop has finished
agent_done The response is complete
agent_error The loop failed

Clients that do not understand these events can ignore them: the answer will arrive in the usual data: chunks.

The number of tool rounds is limited by the Gateway's settings. Tools that must ask a person's permission are never offered through the API, since there is nobody to ask.

When Knowledge Base passages were added to the request automatically, the response will carry the header X-OnVeris-Knowledge-Injected with the number of passages. It will also carry X-Optima-Knowledge-Injected, the same header under its name before the rename, for clients that read that one.

Your Own Tools

Tools you send in tools work as they do with OpenAI: you run them, not the Gateway. They are offered to the model alongside any tools the Gateway has. When the model calls one of yours, the response will end with tool_calls instead of text. For example:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_xyz",
            "type": "function",
            "function": {
              "name": "get_weather",
              "arguments": "{\"city\":\"London\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ]
}

When streaming, each call will arrive as a tool_calls delta, followed by a final chunk with "finish_reason": "tool_calls" and data: [DONE].

Run the function yourself, add the model's message and a tool message with the result to messages, and call the endpoint again for the next step.

  • If the model calls Gateway tools and yours in the same step, the Gateway will run its own first, then return only the calls to yours.
  • The Gateway's results are not part of your conversation. When you come back with your tool results, the model may call one of the Gateway's tools again.
  • When you call an Assistant, your tools are only offered if the Assistant allows API callers' own tools. Otherwise they will be left out. See Assistants.

Controlling The Gateway's Tool Loop

You can control whether the Gateway runs its tools for a request with a header or with tool_choice:

Header or field Effect
X-OnVeris-Agent: false Do not use the Gateway's tools for this request. Your own tools are still offered
X-OnVeris-Agent: true Use them
"tool_choice": "none" Do not use tools at all
"tool_choice": "required" Use the Gateway's tools

A Gateway also reads X-Optima-Agent, the header's name before OnVeris was renamed.

Conversations

/v1/chat/completions keeps no history between requests. Send the whole conversation in messages each time. To have the Gateway keep the history for you, use Responses.

For example:

{
  "model": "fp:9189ab12cd34ef56",
  "messages": [
    { "role": "user", "content": "My name is Alice." },
    { "role": "assistant", "content": "Hello Alice! How can I help you today?" },
    { "role": "user", "content": "What is my name?" }
  ]
}

Context Length

If the messages, plus room for the answer, do not fit the model's context window, the request will be refused rather than shortened:

{
  "error": {
    "message": "…",
    "type": "context_length_exceeded",
    "param": "messages",
    "code": "context_length_exceeded"
  },
  "promptTokens": 40210,
  "contextWindow": 32768
}

Shorten the conversation and try again. Use Listing Models to see each model's ctx_max.

See Errors and Limits for other errors.