A complete GPU-native runtime for production AI.
Inference engine, model serving, identity, policy, observability, evals, and a NIM marketplace — in one platform.
Sub-second inference at any scale.
Every inference request routes through NVIDIA Triton Inference Server and TensorRT-LLM — compiled, quantized, and cached. State is checkpointed; nothing is lost when a GPU node restarts.
- → Bounded retries with backoff and circuit-breakers
- → Per-step caching with semantic deduplication
- → Cross-region replay for compliance + debugging
# nim_pipeline.py await runtime.serve({ model: "meta/llama-3.1-70b-instruct", backend: "triton", policy: "production.guardrails.v2" }); // compiled via TensorRT-LLM
Multi-model pipelines that don't fall over.
A first-class serving graph for prefill / decode / verifier / writer patterns on disaggregated GPU clusters. Compose specialist models with shared memory, tool ACLs, and per-step routing via NVIDIA Dynamo.
Approve before serve, not after.
A declarative DSL — spend caps, allow-/blocklists, escalation paths, two-person rules — evaluated before each inference call. Block prompt injection at the perimeter, not in the model.
policy llm-production { when tool == "nim.serve_request" and request.context_tokens > 100000 require approval from role("platform-team") log reason }
Every token. Every GPU cycle. Every dollar.
OpenTelemetry-native traces with DCGM GPU metrics. Token throughput, GPU utilization, first-token latency, and cost per 1M tokens. Export to Datadog, Grafana, Snowflake.
Ship model updates with confidence, not vibes.
Golden datasets, regression nets, and shadow-eval on live inference traffic. Hook into CI to block deploys when accuracy or latency regresses. Publish your benchmarks — we publish ours.
| Eval | v12 | v13 |
|---|---|---|
| Llama-3.1-70B accuracy | 94.1% | 96.8% |
| TRT-LLM throughput (tok/s) | 4,120 | 6,840 |
| First-token latency (p99) | 180ms | 142ms |
200+ native connectors, NIM endpoints, and MCP tools.
Governed integrations for NVIDIA NIM, Triton, warehouses, CRM, and any MCP server — all policy-bound and observable end to end.
Browse integrationsWhat ships in every plan.
Multi-modal inference
Text, vision, speech, video, and structured data on one serving layer.
Disaggregated serving
Prefill and decode scaled independently across GPU fleets.
Cost-aware routing
Route to the cheapest-capable model with guardrails and latency SLOs.
Human-in-the-loop
Inline review UI for any consequential inference or connector action.
Webhook + REST + MCP
Programmatic embedding for apps, agents, and enterprise tooling.
NeMo fine-tuned specialists
Per-customer models compiled with TensorRT-LLM and benchmarked before deploy.