Open sourceApache 2.0, pre-release. Source on GitHub

Multi-cluster
inference.

An open-source control plane for serving models across Kubernetes GPU clusters, built on NVIDIA Dynamo.

View on GitHub Run it locally

Serving
engines

vLLM

SGLang

TensorRT-LLM

Deployed as
NVIDIA Dynamo

What it does

One model name, served from several clusters.

NVIDIA Dynamo serves a model inside one cluster. Wingsmith decides which clusters run it, tells each one what to run, and routes requests between them.

Cluster placement

Workloads are placed on clusters by device class, capacity, region, and residency.

Endpoints

One model name routes to deployments on several clusters, by weight or in failover order.

Local scheduling

The operator in each cluster reconciles what the plane declares, and the cluster's own scheduler picks the GPUs.

How a model gets served

From a registered cluster to an endpoint.

The plane only ever sends Wingsmith's own objects. This is a deployment as the hub sends it to an operator, which renders it into Kubernetes objects inside the cluster.

Register the cluster
In the console. Registration mints the cluster's enrollment token and renders its values file.
Install the cluster-setup chart
It installs the operator, the agent, the CRDs, the NVIDIA Dynamo platform, and the DCGM exporter. The operator dials out to the hub and starts reporting nodes and devices; the agent dials out to every gateway replica.
Register the weights
Resolving a HuggingFace slug proposes an artifact and writes nothing. Where a vLLM recipe exists, its flags and memory floor come with the draft.
Create a workload
Engine, precision, parallelism, device shape, scale, and placement constraints. The plane picks the clusters and splits the scale into one deployment per cluster.
The operator renders it
Each operator turns its deployment into a DynamoGraphDeployment and reports placement and readiness back up.
Give it an endpoint
One model name for clients. The gateway reads model from the request body and routes to a cluster serving it.

{
  "type": "deployment.desired",
  "seq": 4416,
  "payload": {
    "name": "llama-70b-serving",
    "cluster_id": "clu_01j9x2aaaaaaaaaaaaaaaaaaaa",
    "serving": {
      "engine": "vllm",
      "quantization": "fp8",
      "parallelism": { "tensor": 2, "pipeline": 1, "expert": 1 },
      "context_length": 131072,
      "topology": "disaggregated",
      "router": { "mode": "kv" }
    },
    "min_replicas": 2,
    "max_replicas": 8,
    "placement": {
      "device_class": "h100-80gb-sxm",
      "devices_per_replica": 2,
      "topology": "single_domain"
    },
    "role_replicas": { "prefill": 2, "decode": 4 }
  }
}

A deployment.desired message, trimmed from the contract fixtures in libs/packages/contract.

Control and requests take separate paths.

Both connections are opened from inside the cluster. Requests never pass through the control plane.

Two paths, both opened from the cluster.

The control path and the request path share no process. Every connection is opened from inside the cluster, so no cluster exposes an inbound port.

Illustrative topology. Each cluster can run several engines and models.

No inbound ports. The operator and the agent both dial out, so the plane holds no kubeconfigs and nothing connects into a cluster.

Devices are picked locally. The plane chooses clusters; the scheduler in each cluster chooses GPUs. Read the architecture →

Failure behavior

What keeps working when a part fails.

The plane is not in the request path, so its outages stop changes, not traffic. The gateway is the part to run redundantly.

When	What happens
The control plane is down	Clusters keep serving. Running gateways route from the table they already loaded. Changes wait until the plane is back.
A gateway restarts during a plane outage	It refuses to start without a routing table, so that replica stays out until the plane returns. Persisting the last table is on the roadmap.
An operator restarts during a partition	Desired state lives in CRDs in the cluster and the stream cursor in a StreamCheckpoint, so it resumes from the last acknowledged message.
An endpoint has nowhere to send a request	The gateway answers 503 no_capacity, not 404, so callers can tell a missing model from a model with no capacity.
Every gateway replica is down	Every model is unreachable through the gateway. Run several replicas across zones; regional gateways are not designed yet.

Operating models after they deploy.

Inventory, tracked operations, adoption, and tenancy, in the console and the API.

Device inventory

Per-GPU identity, health, and topology

Identity and health for each device come from DCGM, along with NVLink, NVSwitch, PCIe, and NUMA relationships. Placement reads the same inventory.

Conceptual device topology

Operations

Scale, pause, and drain

Each one is tracked with progress, who started it, why, and how it failed if it did.

RequestedRunningCompleted

Adoption

Models that are already running

They are discovered and observed first. Putting one under management is an explicit step and does not restart it.

How a model is described

Tenancy

Tenants, API keys, and quotas

Tenant endpoints and API keys, and device ceilings per project, applied to one cluster or across the fleet. Serving quotas can be declared but the gateway does not enforce them yet.

Interfaces

Console, CLI, and HTTP API.

The console and hgr are both clients of the control plane's HTTP API. Applications call models through the gateway's OpenAI-compatible API.

The control plane runs locally against a fake fleet, so you can try it without GPUs.

Read the quickstart

Send a request to your configured gateway

curl "$GATEWAY_URL/v1/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-endpoint-name",
    "messages": [
      {"role": "user", "content": "Hello."}
    ],
    "stream": true
  }'

Measurement tools.

Three tools that run without the control plane and keep their raw output: API conformance, load generator comparison, and request-path overhead.

Conformance

Diff an endpoint against its reference provider

Probes an OpenAI-compatible endpoint, stores the raw exchanges, and compares its behavior with the provider whose model it serves.

hgrconform →

Benchmarking

Run four load generators at the same load points

AIPerf, vLLM bench, vLLM sweep, and GuideLLM against one endpoint, reported side by side.

hgrbench →

Request path

Measure the gateway and tunnel overhead

Times the gateway, tunnel, and agent path on one host, without a GPU or a cluster.

hgrtransit →

Tool names shown are from the current pre-release codebase.

Status

Open source,
under Apache 2.0.

Wingsmith is pre-release. The first public release, and everything after it, will be under Apache 2.0.

View on GitHub Read the docs Run it locally

Built and not built

Built

Multi-cluster placement and routing, Dynamo deployments, device inventory, endpoints and quotas, the console, the CLI, and three measurement tools.

Not built

Published images and charts, per-user roles and SSO, batch and training workloads, and a Helm chart for the control plane.

Pre-release. No tagged release, images, or charts yet, and APIs and protocols can change.

Multi-clusterinference.