Executors
An Executor is the process a Node runs to serve a particular AI model. Each model you start has its own. Executors are managed from the Executors panel in a Node's detail view in the Gateway — see Managing Models.
Language models are run by llama.cpp, a widely used open-source inference engine, through a version of its server bundled for your hardware — the backend. See Hardware and GPU.
Executor Lifecycle
When you start a model, the Node will launch its process, which loads the model into GPU memory (or system memory if there is no GPU) and starts answering requests. Loading takes from a few seconds to several minutes, depending on the model's size and the hardware.
Once online, the Executor answers requests from the Gateway until it is stopped or the Node shuts down. When the Node starts again, it will restart the models that were running.
Executor States
An Executor is always in one of the following states:
| State | Meaning |
|---|---|
| Online | The model is loaded and answering requests |
| Starting | The process is launching and loading the model |
| Offline | The Executor is stopped |
| Stopping | The Executor is shutting down |
| Crashed | The Executor failed and the Node has stopped restarting it |
How Long A Model May Take To Start
The Node allows a model time to load in proportion to its size: a minute, plus 15 seconds for every GB of model file, up to 15 minutes. Embedding models are allowed 2 minutes. A model that takes longer will be treated as having failed to start.
Crash Recovery
If a model process crashes, the Node will restart it automatically, waiting a little longer each time. If it crashes 3 times within 10 minutes, the Node will stop trying and show it as Crashed. This stops a faulty or misconfigured model from looping.
To recover, find out why it crashed, fix that, and click start. You can check the following:
- The Node's Log in the Gateway, for the error from the failed process — see Diagnostics
- That the machine has enough memory for the model with its current settings — the memory estimate in its settings shows this. See Executor Settings
- That the model file is complete and not corrupted
- Settings that might not suit the model or the hardware, such as a very large context window or a different backend
Note
If a model's image projector stops it loading, the Node will restart it without the projector, so the model works for text. Its settings will then say it is text-only.
Models That Did Not Finish Starting
If the Node stops — or the machine restarts — while a model is still loading, that model will not be started automatically the next time. This usually means the machine ran out of memory loading it, and starting it again by itself could bring the whole machine down again. The model will show a message saying so. Check its settings, then start it by hand.
Changing Settings
Click the gear icon on an Executor to change its settings. See Executor Settings. Saving while the model is running will restart it with the new settings. Saving while it is stopped will mark it Restart to apply.