Dynamo Runtime — NVIDIA Dynamo Documentation
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Title: Dynamo Runtime — NVIDIA Dynamo Documentation
URL Source: https://docs.nvidia.com/dynamo/archive/0.6.1/development/runtime-guide.html
Published Time: Sat, 08 Nov 2025 00:28:59 GMT
Markdown Content: Skip to main content
Back to top Ctrl+K
latest
latest0.6.10.6.00.5.10.5.00.4.10.4.00.3.20.3.10.3.00.2.10.2.0
Search Ctrl+K
Search Ctrl+K
latest
latest0.6.10.6.00.5.10.5.00.4.10.4.00.3.20.3.10.3.00.2.10.2.0
Table of Contents
Getting Started
Kubernetes Deployment
User Guides
Components
Design Docs
Dynamo Runtime#
A Datacenter Scale Distributed Inference Serving Framework
Rust implementation of the Dynamo runtime system, enabling distributed computing capabilities for machine learning workloads.
Prerequisites#
Install Rust and Cargo using rustup:#
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Build#
cargo build cargo test
Start Dependencies#
Docker Compose#
The simplest way to deploy the pre-requisite services is using docker-compose, defined in deploy/docker-compose.yml.
At the root of the repository:
docker compose -f deploy/docker-compose.yml up -d
This will deploy a NATS.io server and an etcd server used to communicate between and discover components at runtime.
Local (alternate)#
To deploy the pre-requisite services locally instead of using docker-compose above, you can manually launch each:
-
- example:
nats-server -js --trace
- example:
-
etcd server
- follow instructions in etcd installation to start an
etcd-serverlocally
- follow instructions in etcd installation to start an
Run Examples#
When developing or running examples, any process or user that shared your core-services (etcd and nats.io) will be operating within your distributed runtime.
The current examples use a hard-coded namespace. We will address the namespace collisions later.
All examples require the etcd and nats.io pre-requisites to be running and available.
Rust hello_world#
With two terminals open, in one window:
cd examples/hello_world cargo run --bin server
In the second terminal, execute:
cd examples/hello_world cargo run --bin client
which should yield some output similar to:
Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.25s
Running `target/debug/client`
Annotated { data: Some("h"), id: None, event: None, comment: None } Annotated { data: Some("e"), id: None, event: None, comment: None } Annotated { data: Some("l"), id: None, event: None, comment: None } Annotated { data: Some("l"), id: None, event: None, comment: None } Annotated { data: Some("o"), id: None, event: None, comment: None } Annotated { data: Some(" "), id: None, event: None, comment: None } Annotated { data: Some("w"), id: None, event: None, comment: None } Annotated { data: Some("o"), id: None, event: None, comment: None } Annotated { data: Some("r"), id: None, event: None, comment: None } Annotated { data: Some("l"), id: None, event: None, comment: None } Annotated { data: Some("d"), id: None, event: None, comment: None }
Python#
See the README.md for details
The Python and Rust hello_world client and server examples are interchangeable, so you can start the Python server.py and talk to it from the Rust client.
On this page
Privacy Policy | Your Privacy Choices | Terms of Service | Accessibility | Corporate Policies | Product Security | Contact
Copyright © 2024-2025, NVIDIA CORPORATION & AFFILIATES.
Links/Buttons:
- Skip to main content
- NVIDIA Dynamo Documentation
- latest
- 0.6.1
- 0.6.0
- 0.5.1
- 0.5.0
- 0.4.1
- 0.4.0
- 0.3.2
- 0.3.1
- 0.3.0
- 0.2.1
- 0.2.0
- GitHub
- Installation
- Support Matrix
- Examples
- Deployment Guide
- Kubernetes Quickstart
- Detailed Installation Guide
- Dynamo Operator
- Minikube Setup
- Observability (K8s)
- Metrics
- Logging
- Multinode
- Multinode Deployments
- Grove
- Tool Calling
- Multimodality Support
- Finding Best Initial Configs
- Dynamo Benchmarking Guide
- Tuning Disaggregated Performance
- Writing Python Workers in Dynamo
- Observability (Local)
- Metrics Visualization with Prometheus and Grafana
- Health Checks
- Glossary
- Backends
- vLLM
- SGLang
- TensorRT-LLM
- Router
- Planner
- SLA Planner Quick Start
- SLA-Driven Profiling
- SLA-based Planner
- KVBM
- Motivation
- Architecture
- Components
- Design Deep Dive
- Integrations
- KVBM in vLLM
- KVBM in TRTLLM
- LMCache Integration
- Further Reading
- Overall Architecture
- Architecture Flow
- Disaggregated Serving
- Distributed Runtime
- #
- rustup
- docker-compose
- deploy/docker-compose.yml
- NATS.io
- etcd
- Jetstream
- etcd installation
- Privacy Policy
- Your Privacy Choices
- Terms of Service
- Accessibility
- Corporate Policies
- Product Security
- Contact