Abstract / Overview
Cosmos Transfer 2.5, Cosmos Predict 2.5, and Cosmos Reason 2 together form a practical world-modeling stack for physical AI. They address three core needs: scalable realistic data generation, future world prediction, and grounded spatio-temporal reasoning. Isaac GR00T N1.6 complements this stack by converting multimodal observations and language into executable robot actions.
Used together, these systems enable a closed-loop workflow:
Generate and transform simulation data into photorealistic training material
Predict plausible future outcomes under candidate actions
Reason over safety, constraints, and task success
Train, validate, and deploy robot policies
The result is an end-to-end pipeline from simulation to real-world robot execution.
Conceptual Background
Model Roles at a Glance
Cosmos Transfer 2.5
Transforms structured simulation outputs into photorealistic videos using multi-control conditioning such as depth, segmentation, and edges.Cosmos Predict 2.5
Acts as a learned world model that predicts future video trajectories from text, images, or videos.Cosmos Reason 2
Provides spatio-temporal visual reasoning, localization, and structured explanations over images and video sequences.Isaac GR00T N1.6
A vision-language-action robot foundation model that outputs manipulation actions from multimodal inputs.
Why This Stack Works Together
Transfer solves the data realism and scale problem
Predict enables “what happens next” analysis
Reason provides interpretability, constraints, and planning signals
GR00T turns curated data into executable robot behavior
Each stage feeds the next, forming a feedback loop that continuously improves policy quality.
Step-by-Step Walkthrough
Environment Prerequisites
Linux workstation
NVIDIA GPU with sufficient VRAM
CUDA-compatible PyTorch environment
Disk space for model checkpoints and generated assets
Optional: Docker for isolation and reproducibility
Cosmos Transfer 2.5
Installation
git clone [email protected]:nvidia-cosmos/cosmos-transfer2.5.git
cd cosmos-transfer2.5
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv sync
source .venv/bin/activate
uv tool install -U "huggingface_hub[cli]"
hf auth loginRunning Inference
Single GPU
python examples/inference.py \
--params_file assets/robot_example/depth/robot_depth_spec.jsonMulti-GPU
torchrun --nproc_per_node=8 \
-m examples.inference \
--params_file assets/robot_example/vis/robot_vis_spec.json \
--num_gpus=8Control Best Practices
Use control modalities to lock geometry and layout
Use text prompts only for appearance and realism
Reduce prompt creativity when identity preservation matters
Cosmos Predict 2.5
First Rollout
python examples/inference.py \
-i assets/base/robot_pouring.json \
-o outputs/base_video2world \
--inference-type=video2worldInference Modes
Text2World – generate new scenarios
Image2World – extend a single frame
Video2World – predict futures from partial trajectories
Minimal Input JSON
{
"inference_type": "video2world",
"name": "robot_pouring",
"prompt": "A robotic arm pours liquid into a cup on a workbench.",
"input_path": "robot_pouring.mp4"
}Cosmos Reason 2
Installation
git clone https://github.com/nvidia-cosmos/cosmos-reason2.git
cd cosmos-reason2
sudo apt-get install -y curl ffmpeg git git-lfs
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv sync
hf auth loginServing with vLLM
uv run vllm serve nvidia/Cosmos-Reason2-2B \
--allowed-local-media-path "$(pwd)" \
--max-model-len 8192 \
--media-io-kwargs '{"video": {"num_frames": -1}}' \
--reasoning-parser qwen3 \
--port 8000Typical Outputs Used in Robotics
Temporal event localization
Object-level constraints
Failure explanations
Plan step generation
Isaac GR00T N1.6
Repository Setup
git clone --recurse-submodules https://github.com/NVIDIA/Isaac-GR00T
cd Isaac-GR00T
git submodule update --init --recursiveRecommended Workflow
Collect demonstrations
Convert to LeRobot format
Validate zero-shot policy
Fine-tune on curated data
Benchmark in simulation
Deploy to robot controller
Why N1.6 Matters
Improved VLM backbone
Larger action diffusion head
Expanded teleoperation data
Better cross-embodiment transfer
End-to-End Integration Pipeline
Workflow Manifest Example
{
"run_name": "robotics-world-modeling-to-policy-v1",
"transfer25": { "model": "Cosmos-Transfer2.5" },
"predict25": { "inference_type": "video2world" },
"reason2": { "serve_port": 8000 },
"groot_n16": { "finetune": true }
}Use Cases
Simulation-to-real data scaling
Policy evaluation through predictive rollouts
Reasoning-guided failure mining
Embodiment adaptation for new robots
Limitations and Considerations
World model rollouts are probabilistic, not physics-ground-truth
Long-context reasoning increases memory pressure
Control misconfiguration is the most common failure source
Model checkpoints may require license acceptance
Common Fixes
Label drift after Transfer
Increase control strength and reduce prompt dominanceUnstable Predict rollouts
Shorten horizon and increase conditioning framesReason 2 OOM errors
Reduce max context length and video FPSGR00T underperformance
Validate data schema before tuning hyperparameters
FAQs
Do I need all four models?
No. Start with Transfer + GR00T, then add Predict and Reason as complexity grows.What is the fastest proof of value?
Fine-tune GR00T on Transfer-generated photoreal data and compare against zero-shot performance.How should this be deployed?
Transfer and Predict as offline batch jobs, Reason as a service, GR00T close to the robot controller.
Conclusion
Cosmos Transfer 2.5, Cosmos Predict 2.5, Cosmos Reason 2, and Isaac GR00T N1.6 together form a complete physical AI pipeline. Transfer scales realism, Predict explores futures, Reason enforces understanding, and GR00T executes actions. When connected through reproducible specs and validated datasets, this stack enables robust robot learning from simulation to real-world deployment.
Join the conversation! Your thoughts help the community grow.