> ## Documentation Index > Fetch the complete documentation index at: https://overmind-b84ae13c-codex-workshop-queue-capacity.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Inference > Call your fine-tuned models through an OpenAI-compatible endpoint — deployment lifecycle, the capability alias, request fields, error codes, and the Console Inference page. Every successful [training run](/models/training) registers a deployment and serves it from Overmind's GPU pool through an OpenAI-compatible chat completions endpoint. You call it with your existing client and an Overmind API key; the model id is either the deployment's reference or a capability alias that follows whichever model you have made live. The Console model list shows deployments from training runs. Base-model deployments used internally for evaluations are excluded from the list and its counts; they remain available to evaluation jobs. ## How a model is served A deployment moves through `queued → deploying → warming → ready`, or ends `failed`, `deleting`, `deleted`. Two shapes: * A dense or explicitly supported mixture-of-experts model trained on Modal with LoRA is served as an **adapter on a shared BF16 base**; deployments of the same base share one process and skip the `quantizing` stage. * Everything else — full fine-tunes, other mixture-of-experts models, models whose family has no adapter support — is **merged and quantised into a private checkpoint** (FP8 where the base supports it) and served from its own vLLM process. Each process scales to zero when idle: two minutes without traffic for most models, fifteen for the largest. The first request after that cold-boots the model in two to seven minutes; the response is held open with an idle ping every 15 seconds so the connection survives. Deployment progress reports base-weight preparation, checkpoint preparation, registration, and boot-and-verification. Remote operation handles are saved: a worker restart reconnects to the same work. One controller checks training outputs and evaluation baselines every 15 seconds; an abandoned worker claim expires after 45 seconds. Confirmed failures retry after 30 and 60 seconds, with a four-hour overall deadline. Transport errors reconnect to the existing operation. If submission succeeded but its acknowledgement was lost, deployment stops rather than risking duplicate GPU work; remote state must be resolved before retrying. The last error, retry time, attempt, and deadline are available through the deployment API and MCP resource. Deployment failure does not discard or repeat successful training. The checkpoint is retained and its dependent evaluation is marked failed. An explicit deployment retry starts a new bounded attempt and reuses prepared artifacts where available. Before-training evaluations report baseline deployment progress on the waiting training run. If that deployment fails, the evaluation is marked failed and the next training poll stops submission with the deployment error, even while remote cancellation is awaiting confirmation. ## Calling your model Point any OpenAI-compatible client at `https://api.overmindlab.ai/api/v1` with your API key (`Authorization: Bearer ` or `X-Api-Key: `). The model page’s **View API snippet** and coding-agent prompt use the current server’s URL, default to the capability alias and offer an explicit version pin. Reuse a project key in `OVERMIND_API_KEY`, or open the existing key-creation flow from the snippet dialog. ```python theme={null} import os from openai import OpenAI client = OpenAI(base_url="https://api.overmindlab.ai/api/v1", api_key=os.environ["OVERMIND_API_KEY"]) response = client.chat.completions.create( model="overmind/", messages=[{"role": "user", "content": "Classify this ticket: ..."}], ) print(response.choices[0].message.content) ``` The SDK ships the same surface without the OpenAI dependency: `overmind.Client()` reads `OVERMIND_API_KEY` and `OVERMIND_API_URL`, and exposes `chat.completions.create(...)` (streaming with `stream=True`), `models.list(status=)`, `models.get(id)` and `models.delete(id)`. Two ids work in `model`: * **The deployment reference** — `ft--`, the deployment's `model_id` on its page — pins one model. * **The capability alias** `overmind/` resolves to the capability's `active_model`. **Make live** on a deployment retargets it; the next request hits the new model. Model ids under `anthropic/`, `deepseek/`, `google/`, `meta-llama/`, `openai/`, `qwen/` and `x-ai/` are frontier models the same endpoint forwards to OpenRouter, so a repository can switch between frontier and fine-tuned models by changing the id alone. They are listed only when the API has an OpenRouter key configured. ### Request fields Serving context includes both the input and generated output. Training sequence length does not determine deployment capacity. Requests reserve an explicit positive output budget (8,192 tokens when omitted); an input that cannot fit alongside that reservation is rejected instead of silently reducing output space. Prompt truncation is not supported. `max_completion_tokens` is accepted as an alternative to `max_tokens` and takes precedence when both are supplied. Inspect each choice's `finish_reason`: `length` indicates an incomplete answer, even when HTTP transport succeeded. Streaming responses retain this finish reason. MCP `run_inference` returns `finish_reason` and `truncated`; `content_clipped` separately identifies its response-size bound. If capacity is insufficient, reduce the request or deploy with a larger supported serving context. `POST /api/v1/chat/completions` accepts `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `frequency_penalty`, `presence_penalty`, `tools`, `tool_choice`, `response_format`, `stream`, `stream_options.include_usage`, `reasoning_effort`, `include_reasoning` and `chat_template_kwargs`. Unknown fields are ignored. * **Reasoning.** For families that emit chain-of-thought, `include_reasoning` defaults to `false` and `reasoning_effort` to the family's default; pass them explicitly to override. `chat_template_kwargs` is merged over the family's defaults and forwarded to the chat template. * **`max_tokens`.** Some families spend part of the budget on a preamble that cannot be turned off; a value below the family's floor is raised to it so the answer is not empty. * **Streaming.** `"stream": true` returns SSE `data:` events; the idle ping is an SSE comment line (`: `). * **Non-streaming to a deployed model.** A result or failure within 15 seconds returns a normal JSON response with its success or error status. Longer requests send newline keepalives every 15 seconds. If inference fails after keepalives commit the headers, the response terminates without a complete JSON body, producing a transport error in the OpenAI client. SSE failures carry an error frame. `X-Request-ID` correlates deployed-model requests with recorded failures. ### Error codes | Status | Trigger | | - | - | | `400` | Malformed body: not a JSON object, `model` or `messages` missing, numeric fields of the wrong type, an alias that is not `overmind/`; on `DELETE`, a frontier id or an alias | | `402` | Insufficient credits | | `404` | No deployment matches the id in your projects; the alias's capability has no `active_model`; `GET` on a frontier id | | `409` | `DELETE` on a deployment already `deleting` or `deleted` | | `429` | OpenRouter rate limit on a frontier model | | `502` | OpenRouter or the serving backend returned an error | | `503` | The deployment or the alias's target is not `ready`; a frontier id was requested and no OpenRouter key is configured | ## Managing deployments ```bash theme={null} curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/v1/models" # ?status=ready by default; ?status=all curl -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/v1/models/{model_id}" curl -X DELETE -H "X-Api-Key: $OVERMIND_API_KEY" "https://api.overmindlab.ai/api/v1/models/{model_id}" ``` `GET /api/v1/models` lists `ready` deployments by default; `?status=` accepts any deployment status or `all`, and anything else is `400`. With `ready` or `all` the list also carries one `overmind/` row per capability whose active model is ready, and the OpenRouter frontier models when a key is configured. `GET .../{model_id}` accepts a deployment reference or an alias. `DELETE` on a deployment marks it `deleting`, removes it from the serving backend, marks it `deleted` and returns `{"id": ..., "object": "model", "deleted": true}`; the row stays, and a failed backend removal leaves it `failed` with a `502`. The Console uses the project API, `/api/deployed-models/`, for the rest: | Action | Effect | | - | - | | `POST .../{id}/deploy/` | Re-runs registration for the deployment's training job | | `POST .../{id}/retry/` | Resets a `failed` or `deleted` deployment to `queued` and re-runs registration; the training job must be `succeeded` or `deploying`. The **Retry** / **Deploy** button on the model page's banner | | `POST .../{id}/undeploy/` | Removes the model from the serving backend and marks it `deleted`; weights are kept, so `retry/` brings it back | | `DELETE .../{id}/` | Same teardown from the Console's **Delete model** | | `GET .../{id}/metrics/` | Request count, prompt and completion tokens, median and p95 warm latency, median throughput, estimated cold-request count and latency, failed requests, end-to-end p50/p95, latest failure request id, cost this month and total, and the savings against the capability's original model priced at its OpenRouter rate | | `GET .../{id}/activity/` | Requests, failures, tokens, median warm throughput, and end-to-end p50/p95 latency per `minute`, `hour` or `day` bucket | | `GET .../{id}/live/` | Recent successful activity and warming signals; worker availability remains unknown when no per-deployment measurement exists | | `GET .../{id}/checkpoints/` | A presigned download link for the trained weights, valid for one hour | **Make live** is `PATCH /api/capabilities/{id}/` with `active_model` set to a ready deployment. This starts a saved activation: **Check deployment → Wake & verify → Switch routing**. The response keeps the existing `active_model` until the verification request succeeds. Poll the capability’s `activation` field for progress, including after navigating away. Failures identify the stage and offer **Retry activation**. The Console shows **Waiting for first application request** after activation, then the timestamp of the first successful application API-key request to the alias. Verification probes, internal evaluations and requests pinned to a version do not confirm alias integration. Making a model live wakes it once; idle scale-down still applies. Through MCP, call `set_active_model` and poll its `model_activation` receipt with `get_job`. Clearing `active_model` cancels an in-progress switch and removes alias routing. The CLI's `overmind model download-checkpoint ` fetches the weights through `checkpoints/`. ## The Console Inference page Inference page listing deployed models The list shows **Model**, **Base model**, **Capability**, **Training job**, **Reference**, **Status**, **Requests**, **Tokens**, **Throughput**, **Latency** and **Last active**; latency and throughput are medians over warm calls. Status shows the deployment status until it is `ready`; a ready model shows its serving state from `live/` — **Warm** when its worker pool has running containers or it served successful traffic in the last 90 seconds, **Warming** while a cold boot is in progress, **Asleep** when its measured pool has no containers, and **Unknown** only when worker measurements are unavailable and there is no recent serving signal. The list and detail page refresh this state every five seconds; checking status does not wake the model. The separate **Live model** badge identifies the capability’s selected route, independently of worker warmth. The model page focuses on **Inference performance**: request volume, success rate, end-to-end p50 and p95 response time, median warm generation speed, and estimated serving cost. The reporting period applies to both headline metrics and charts: **Last hour**, **Last 24 hours**, **Last 7 days**, **Last 30 days** or **All time**. **Application traffic** includes API-key requests to either the capability alias or a pinned model ID; **All traffic** also includes internal calls and requests with an unknown source. Metrics refresh every 15 seconds while the page is open. The charts show request traffic and failures, response-time percentiles, and generation speed. Empty traffic buckets are zero; missing timing measurements remain gaps. Failed calls do not contribute to latency or speed distributions. A single empty state replaces the charts when the selected period has no requests. Refresh errors are visible and do not turn into zero traffic. The latest failure includes its request ID. The serving row retains the capability alias, **Make live**, stage-based activation progress and **Copy prompt**, which copies coding-agent instructions for the capability alias directly to the clipboard. **Deployment details** contains base model, training job, GPU, context, precision, weights download and the guarded delete action. A failed or deleted deployment retains **Retry** or **Deploy**. Both `metrics/` and `activity/` accept `period=1h|24h|7d|30d|all` and `source=application|all`; both default to `all`. `activity/` also accepts `granularity=minute|hour|day`. Percentile calculations use the most recent 5,000 measured successful requests in the selected population (per bucket for charts). Estimated cost reports how many requests have a recorded cost when coverage is incomplete. Inference model detail page ## Good to know * Every request through the endpoint is recorded as an inference call and traced under the project, so a live fine-tuned model feeds the datasets for its next round of training. * Merged checkpoints are quantised during weight preparation; adapter deployments use a shared BF16 base. * A retry after `failed` validates base weights again and reuses cached checkpoint preparation. It never restarts training. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.