Skip to main content
Every successful training run registers a deployment and serves it from Overmind’s GPU pool through an OpenAI-compatible chat completions endpoint. You call it with your existing client and an Overmind API key; the model id is either the deployment’s reference or a capability alias that follows whichever model you have made live. The Console model list shows deployments from training runs. Base-model deployments used internally for evaluations are excluded from the list and its counts; they remain available to evaluation jobs.

​
How a model is served

A deployment moves through queued → deploying → warming → ready, or ends failed, deleting, deleted. Two shapes:
  • A dense or explicitly supported mixture-of-experts model trained on Modal with LoRA is served as an adapter on a shared BF16 base; deployments of the same base share one process and skip the quantizing stage.
  • Everything else — full fine-tunes, other mixture-of-experts models, models whose family has no adapter support — is merged and quantised into a private checkpoint (FP8 where the base supports it) and served from its own vLLM process.
Each process scales to zero when idle: two minutes without traffic for most models, fifteen for the largest. The first request after that cold-boots the model in two to seven minutes; the response is held open with an idle ping every 15 seconds so the connection survives. Deployment progress reports base-weight preparation, checkpoint preparation, registration, and boot-and-verification. Remote operation handles are saved: a worker restart reconnects to the same work. One controller checks training outputs and evaluation baselines every 15 seconds; an abandoned worker claim expires after 45 seconds. Confirmed failures retry after 30 and 60 seconds, with a four-hour overall deadline. Transport errors reconnect to the existing operation. If submission succeeded but its acknowledgement was lost, deployment stops rather than risking duplicate GPU work; remote state must be resolved before retrying. The last error, retry time, attempt, and deadline are available through the deployment API and MCP resource. Deployment failure does not discard or repeat successful training. The checkpoint is retained and its dependent evaluation is marked failed. An explicit deployment retry starts a new bounded attempt and reuses prepared artifacts where available. Before-training evaluations report baseline deployment progress on the waiting training run. If that deployment fails, the evaluation is marked failed and the next training poll stops submission with the deployment error, even while remote cancellation is awaiting confirmation.

​
Calling your model

Point any OpenAI-compatible client at https://api.overmindlab.ai/api/v1 with your API key (Authorization: Bearer <key> or X-Api-Key: <key>). The model page’s View API snippet and coding-agent prompt use the current server’s URL, default to the capability alias and offer an explicit version pin. Reuse a project key in OVERMIND_API_KEY, or open the existing key-creation flow from the snippet dialog.
The SDK ships the same surface without the OpenAI dependency: overmind.Client() reads OVERMIND_API_KEY and OVERMIND_API_URL, and exposes chat.completions.create(...) (streaming with stream=True), models.list(status=), models.get(id) and models.delete(id). Two ids work in model:
  • The deployment reference — ft-<first 8 chars of the job id>-<base model>, the deployment’s model_id on its page — pins one model.
  • The capability alias overmind/<capability-uuid> resolves to the capability’s active_model. Make live on a deployment retargets it; the next request hits the new model.
Model ids under anthropic/, deepseek/, google/, meta-llama/, openai/, qwen/ and x-ai/ are frontier models the same endpoint forwards to OpenRouter, so a repository can switch between frontier and fine-tuned models by changing the id alone. They are listed only when the API has an OpenRouter key configured.

​
Request fields

Serving context includes both the input and generated output. Training sequence length does not determine deployment capacity. Requests reserve an explicit positive output budget (8,192 tokens when omitted); an input that cannot fit alongside that reservation is rejected instead of silently reducing output space. Prompt truncation is not supported. max_completion_tokens is accepted as an alternative to max_tokens and takes precedence when both are supplied. Inspect each choice’s finish_reason: length indicates an incomplete answer, even when HTTP transport succeeded. Streaming responses retain this finish reason. MCP run_inference returns finish_reason and truncated; content_clipped separately identifies its response-size bound. If capacity is insufficient, reduce the request or deploy with a larger supported serving context. POST /api/v1/chat/completions accepts model, messages, temperature, max_tokens, top_p, frequency_penalty, presence_penalty, tools, tool_choice, response_format, stream, stream_options.include_usage, reasoning_effort, include_reasoning and chat_template_kwargs. Unknown fields are ignored.
  • Reasoning. For families that emit chain-of-thought, include_reasoning defaults to false and reasoning_effort to the family’s default; pass them explicitly to override. chat_template_kwargs is merged over the family’s defaults and forwarded to the chat template.
  • max_tokens. Some families spend part of the budget on a preamble that cannot be turned off; a value below the family’s floor is raised to it so the answer is not empty.
  • Streaming. "stream": true returns SSE data: events; the idle ping is an SSE comment line (: ).
  • Non-streaming to a deployed model. A result or failure within 15 seconds returns a normal JSON response with its success or error status. Longer requests send newline keepalives every 15 seconds. If inference fails after keepalives commit the headers, the response terminates without a complete JSON body, producing a transport error in the OpenAI client. SSE failures carry an error frame. X-Request-ID correlates deployed-model requests with recorded failures.

​
Error codes

​
Managing deployments

GET /api/v1/models lists ready deployments by default; ?status= accepts any deployment status or all, and anything else is 400. With ready or all the list also carries one overmind/<capability-uuid> row per capability whose active model is ready, and the OpenRouter frontier models when a key is configured. GET .../{model_id} accepts a deployment reference or an alias. DELETE on a deployment marks it deleting, removes it from the serving backend, marks it deleted and returns {"id": ..., "object": "model", "deleted": true}; the row stays, and a failed backend removal leaves it failed with a 502. The Console uses the project API, /api/deployed-models/, for the rest: Make live is PATCH /api/capabilities/{id}/ with active_model set to a ready deployment. This starts a saved activation: Check deployment → Wake & verify → Switch routing. The response keeps the existing active_model until the verification request succeeds. Poll the capability’s activation field for progress, including after navigating away. Failures identify the stage and offer Retry activation. The Console shows Waiting for first application request after activation, then the timestamp of the first successful application API-key request to the alias. Verification probes, internal evaluations and requests pinned to a version do not confirm alias integration. Making a model live wakes it once; idle scale-down still applies. Through MCP, call set_active_model and poll its model_activation receipt with get_job. Clearing active_model cancels an in-progress switch and removes alias routing. The CLI’s overmind model download-checkpoint <deployment-id> fetches the weights through checkpoints/.

​
The Console Inference page

Inference page listing deployed models

Inference: every deployed model with its lineage, status, and request metrics.

The list shows Model, Base model, Capability, Training job, Reference, Status, Requests, Tokens, Throughput, Latency and Last active; latency and throughput are medians over warm calls. Status shows the deployment status until it is ready; a ready model shows its serving state from live/ — Warm when its worker pool has running containers or it served successful traffic in the last 90 seconds, Warming while a cold boot is in progress, Asleep when its measured pool has no containers, and Unknown only when worker measurements are unavailable and there is no recent serving signal. The list and detail page refresh this state every five seconds; checking status does not wake the model. The separate Live model badge identifies the capability’s selected route, independently of worker warmth. The model page focuses on Inference performance: request volume, success rate, end-to-end p50 and p95 response time, median warm generation speed, and estimated serving cost. The reporting period applies to both headline metrics and charts: Last hour, Last 24 hours, Last 7 days, Last 30 days or All time. Application traffic includes API-key requests to either the capability alias or a pinned model ID; All traffic also includes internal calls and requests with an unknown source. Metrics refresh every 15 seconds while the page is open. The charts show request traffic and failures, response-time percentiles, and generation speed. Empty traffic buckets are zero; missing timing measurements remain gaps. Failed calls do not contribute to latency or speed distributions. A single empty state replaces the charts when the selected period has no requests. Refresh errors are visible and do not turn into zero traffic. The latest failure includes its request ID. The serving row retains the capability alias, Make live, stage-based activation progress and Copy prompt, which copies coding-agent instructions for the capability alias directly to the clipboard. Deployment details contains base model, training job, GPU, context, precision, weights download and the guarded delete action. A failed or deleted deployment retains Retry or Deploy. Both metrics/ and activity/ accept period=1h|24h|7d|30d|all and source=application|all; both default to all. activity/ also accepts granularity=minute|hour|day. Percentile calculations use the most recent 5,000 measured successful requests in the selected population (per bucket for charts). Estimated cost reports how many requests have a recorded cost when coverage is incomplete.
Inference model detail page

A model page: identity and metrics on top, request activity underneath.

​
Good to know

  • Every request through the endpoint is recorded as an inference call and traced under the project, so a live fine-tuned model feeds the datasets for its next round of training.
  • Merged checkpoints are quantised during weight preparation; adapter deployments use a shared BF16 base.
  • A retry after failed validates base weights again and reuses cached checkpoint preparation. It never restarts training.