Skip to content

strands-cosmos: Physical AI Reasoning with NVIDIA Cosmos on Strands Agents 🌌

Published: March 26, 2026

What if your AI agent could watch a dashcam video and reason about traffic safety? Or look through a robot's camera and plan the next action? Or analyze whether a physics simulation looks plausible?

That's strands-cosmos: NVIDIA's Cosmos-Reason2 vision-language models, packaged as a Strands Agents model provider — bringing physical world understanding to any agent.

The Problem

Large language models are great at text. But the physical world is video, images, motion, and causality. Robots need to see. Self-driving cars need to understand. And agents need to reason about physics.

NVIDIA's Cosmos-Reason2 models are built for exactly this — trained on massive physical-world datasets for video understanding, driving analysis, embodied reasoning, and physics plausibility. But using them in an agent workflow meant writing custom inference pipelines.

strands-cosmos wraps it all in the Strands Model interface. Three lines, and your agent can see.

Quick Start

from strands import Agent
from strands_cosmos import CosmosVisionModel

model = CosmosVisionModel(model_id="nvidia/Cosmos-Reason2-2B")
agent = Agent(model=model)

# Video understanding
agent("Caption in detail: <video>dashcam.mp4</video>")

# Image reasoning
agent("<image>robot_view.jpg</image> What can be the next immediate action?")

# Physics reasoning (text-only works too)
agent("What happens when a ball rolls off a table?")

graph LR
    V[🎬 Video] -->|frames @ 4fps| M[Cosmos-Reason2]
    I[🖼️ Image] -->|vision tokens| M
    T[📝 Text] -->|prompt| M
    M -->|chain-of-thought| R[Physical AI Reasoning]
    R --> D[🚗 Driving Analysis]
    R --> E[🤖 Robot Planning]
    R --> P[⚛️ Physics Understanding]

    style M fill:#76b900,color:#fff,stroke:#333,stroke-width:3px

What Cosmos-Reason2 Can Do

These aren't generic VLMs — they're specialized for the physical world:

Capability Example
🚗 Driving Analysis Traffic hazards, navigation from dashcam
🤖 Robot Planning Next-action prediction, 2D trajectory
🎬 Video Captioning Temporal-spatial descriptions
⚛️ Physics Reasoning Object permanence, causality, plausibility
🔍 2D Grounding Bounding box localization
📍 Temporal Localization Event timestamps in video
🧠 Chain-of-Thought <think> reasoning before answers

Chain-of-Thought Reasoning

Enable reasoning=True and Cosmos generates its thinking process before answering:

model = CosmosVisionModel(reasoning=True)
agent = Agent(model=model)

agent("<video>intersection.mp4</video> Is this safe to proceed?")

Output: 

<think>
The video shows the vehicle approaching an intersection. There is a 
pedestrian on the crosswalk to the right who appears to be mid-crossing.
The traffic light is green, but the pedestrian has right of way...
</think>

No, it is not safe to proceed. A pedestrian is actively crossing 
the street from the right side. The vehicle should yield and wait 
for the pedestrian to clear the crosswalk before proceeding.

This isn't just classification — it's reasoning about physics and safety.

Three Ways to Use It

1. As a Strands Model (full agent)

from strands import Agent
from strands_cosmos import CosmosVisionModel

model = CosmosVisionModel(
    model_id="nvidia/Cosmos-Reason2-2B",
    reasoning=True,
    fps=4,
    params={"max_tokens": 4096, "temperature": 0.6},
)
agent = Agent(model=model)
agent("Caption: <video>warehouse_robot.mp4</video>")

2. As a Tool (in any agent)

Use Cosmos as a vision tool inside a Bedrock/OpenAI agent:

from strands import Agent
from strands_cosmos import cosmos_vision_invoke

agent = Agent(tools=[cosmos_vision_invoke])
agent("Analyze this dashcam for safety: /path/to/video.mp4")

Your Claude/GPT agent orchestrates, Cosmos does the seeing.

3. Built-in Task Prompts

Pre-built prompts for common physical AI tasks:

from strands_cosmos.cosmos_vision_model import TASK_PROMPTS

# Available:
#   caption, embodied_reasoning, driving, causal,
#   temporal_localization, 2d_grounding, robot_cot,
#   describe_anything, mvp_bench

Architecture

strands_cosmos/
├── cosmos_model.py          # Text-only (Strands Model interface)
├── cosmos_vision_model.py   # Vision: video + image + text
├── fix_cublas.py            # Jetson CUBLAS compatibility fix
└── tools/
    ├── cosmos_invoke.py         # Text inference tool
    └── cosmos_vision_invoke.py  # Vision inference tool

~1,400 lines total. Two model providers (text-only and vision), two tools, and a hardware compatibility fix.

flowchart TD
    Agent[Strands Agent]
    Agent --> CosmosVision[CosmosVisionModel]
    Agent --> CosmosTool[cosmos_vision_invoke tool]

    CosmosVision --> HF[HuggingFace Transformers]
    CosmosTool --> CosmosVision

    HF --> GPU[NVIDIA GPU]
    GPU --> Thor[Jetson Thor 132GB]
    GPU --> Desktop[A100 / H100 / 4090]
    GPU --> Orin[Jetson Orin 32GB]

    style Agent fill:#e1f5ff,stroke:#333
    style CosmosVision fill:#76b900,color:#fff,stroke:#333,stroke-width:3px
    style Thor fill:#ff9,stroke:#333

Models

Model GPU Memory Best For
Cosmos-Reason2-2B 24GB Edge deployment, Jetson
Cosmos-Reason2-8B 32GB Higher accuracy, desktop

Both are based on Qwen3-VL architecture, fine-tuned on NVIDIA's physical-world datasets.

Running on Jetson

This was battle-tested on Jetson AGX Thor (132GB unified memory, SM 11.0). One gotcha: PyTorch's pip-bundled CUBLAS doesn't support Thor's GPU architecture. We ship a fix:

pip install strands-cosmos strands-agents

# Fix CUBLAS (auto-detects, safe on any platform)
strands-cosmos-fix-cublas

What the CUBLAS fix does

PyTorch ships its own libcublas.so which may not support Jetson SM architectures. The fix:

  1. Backs up torch's bundled CUBLAS
  2. Copies system CUBLAS from JetPack
  3. Verifies with a quick torch.mm test

Reversible: strands-cosmos-fix-cublas --revert

Why This Matters

Physical AI is the Next Frontier

Text LLMs solved language. Vision models solved images. But understanding the physical world — motion, causality, physics, safety — requires specialized reasoning. Cosmos-Reason2 is NVIDIA's bet on this future.

The Strands Model Interface Scales

The same Model interface that talks to Claude Opus also runs a 2B-parameter vision model on a Jetson. Same Agent(). Same tools=[]. Different physics.

flowchart LR
    Interface[Strands Model Interface]
    Interface --> Bedrock[Amazon Bedrock]
    Interface --> Anthropic[Claude]  
    Interface --> MicroGPT[🧠 MicroGPT<br/>815 lines Python]
    Interface --> Cosmos[🌌 Cosmos-Reason2<br/>Physical AI VLM]

    style Cosmos fill:#76b900,color:#fff,stroke:#333,stroke-width:3px
    style MicroGPT fill:#ff9,stroke:#333

Agents That See

An agent with cosmos_vision_invoke as a tool can:

  • Watch security camera feeds and alert on anomalies
  • Analyze dashcam footage for insurance claims
  • Guide a robot through a task by understanding its camera view
  • Verify physics simulations for plausibility

Demo: Dashcam Safety Analysis

Running on Jetson AGX Thor with Chain-of-Thought reasoning:

Strands Cosmos Demo

The agent watches dashcam video frame-by-frame, reasons about traffic conditions, identifies hazards, and provides safety analysis — all in real-time on edge hardware.

Install

pip install strands-cosmos strands-agents

Requirements: Python ≥3.10, NVIDIA GPU (24GB+ for 2B, 32GB+ for 8B)

Your agent can now see the physical world. What will it understand?