Measurement tools
Conformance, load,
and request-path tools.
Three tools that run independently of the control plane. Each keeps its raw output, so a finding can be traced back to what was sent and what came back.
hgrconformAPI conformance against a reference provider.
Point probes at any OpenAI-compatible endpoint, record what it did, and diff it against a reference provider.
If you serve someone else's model, the requirement is usually that it behaves like the original provider: whether thinking is on by default, where usage appears in a stream, whether an out-of-range parameter returns 400 or 200. The spec is the provider's behavior, not a fixed answer.
So probes don't assert. A probe records what the endpoint did, and the result is the diff against the reference.
- Vendor knowledge lives in the profile, never in a probe.Thinking syntax, cached-token field paths, and the claimed context window are profile fields.
- Every exchange is stored verbatim.Any finding can be traced back to the raw request and response.
- No sampling parameters unless the probe is about sampling.Some providers pin them, so sending one a probe doesn't need would hide the default.
just conformance <profile> # observe one endpoint
just conformance <profile> --reference <provider> # and diff it against the provider
just conformance-probes # what it knows how to ask
just conformance-diff # re-diff the latest stored runIt knows nothing about the control plane: a Wingsmith gateway is one endpoint it can check, and a provider's own API is another. The console's benchmark interface does not drive it yet.
hgrbenchFour load generators at the same load points.
Results are reported side by side, so each tool's numbers can be checked against the others.
| Harness | What it is for | Load model |
|---|---|---|
aiperf | NVIDIA AIPerf. The reference, because the plane's own benchmarks module pins the same image and version. | Closed loop, N in flight |
vllm-bench | vLLM's bench serve. The only one here that can hold an arrival rate instead of a concurrency. | Open or closed loop |
vllm-sweep | vLLM's own parameter sweep, writing upstream's own summary and plots. | Ladder, one process |
guidellm | GuideLLM. Multi-point by design, with a sweep profile that self-calibrates. | Ladder or self-calibrating sweep |
Three stages of a run
- 01MatrixEvery model, load point, harness, and repeat, one cell at a time, so a difference is about the tools.
- 02LaddersHarnesses that can walk a whole curve in one process do so, with no cold start per rung.
- 03Saturation searchClimb until p90 TTFT crosses the SLA or a request fails. Opt-in, and it needs --hard.
Runs are described by committed TOML plans, not flags, so what got measured is reviewable in a diff. Everything runs serially, because two load generators against one deployment would skew each other. Original exports are kept and normalized through explicit field maps, without averaging away disagreement between tools.
just bench-suite smoke # about 10 minutes, safe any time of day
just bench-suite deep --hard # about 2 hours, includes the saturation search
just bench-report # re-render the latest runResults depend on the workload, model, hardware, and measurement configuration. The project ships a method note to read before quoting a number, and makes no blanket performance claim.
hgrtransitRequest-path overhead.
Client to gateway to tunnel to agent to frontend, measured without a GPU, a cluster, or a model.
It answers two questions in order: where the bottleneck is, and how much one gateway replica can carry. The benchmarking tool measures an engine on real hardware; this one measures the gateway, tunnel, and agent, with a fake frontend in place of the engine.
| Mode | What is real | Blast radius |
|---|---|---|
synthetic | Gateway, agent, tunnel, and both reverse proxies | This host only |
path | Gateway, agent, and tunnel, in front of a frontend you run | This host only |
live | Everything, including a real engine | Offered load against a shared serving path |
The comparable number is the overhead over baseline, which only synthetic mode can produce, because only there is the frontend's own contribution known. Synthetic throughput is a ceiling for the transport, never a capacity for the product.
Live mode makes real completions on a shared path. The default ramp is 1, 2, 4, and any stage above a concurrency of 8 is refused. By default it is a small probe, not a load test.
just transit # synthetic ramp through the real gateway, tunnel and agent
just transit-live <url> <model> # against a deployed gateway, small by defaultStatus
Open source,
under Apache 2.0.
Wingsmith is pre-release. The first public release, and everything after it, will be under Apache 2.0.
Built and not built
Built
Multi-cluster placement and routing, Dynamo deployments, device inventory, endpoints and quotas, the console, the CLI, and three measurement tools.
Not built
Published images and charts, per-user roles and SSO, batch and training workloads, and a Helm chart for the control plane.
Pre-release. No tagged release, images, or charts yet, and APIs and protocols can change.