Skip to content

🔌 Glue Is the Bottleneck, Not the Model: O(M·N) → O(M+N) via One Policy ABC

Published: June 28, 2026

🎧 First in first, background music.

TL;DR

The robotics community keeps shipping bigger models (Pi0, Pi0.5, GR00T N1.5, MolmoAct2, SmolVLA, RT-2, OpenVLA) and bigger simulators (MuJoCo, Isaac Sim, Genesis, Newton). What it does not keep shipping is the connective tissue between them. As of mid-2026, every new (robot, policy, simulator) triple still requires bespoke glue. This is the O(M·N) glue problem: with M policies and N robots/simulators, the world ends up writing M·N integration adapters, each one a quiet source of seam corruption, frame-rate drift, and silent dataset fabrication.

This post argues — and proves with code — that the fix is mundane and well-known to every backend engineer: one abstract base class, one disciplined contract, and the cost drops to O(M+N). We show how strands-labs/robots collapses the matrix using a single Policy ABC, a Robot tool, and a backend-agnostic Simulation interface. We also surface the failure modes we found along the way — including a particularly nasty one where the control loop bypasses the policy's real-time chunking machinery on hardware — and propose the missing async runtime that turns RTC from "seam removal" into actual latency masking.

1. The Problem, Stated Crisply

You have:

  • M policies: Pi0, Pi0.5, Pi0-FAST, SmolVLA, MolmoAct2, GR00T N1.5, ACT, cuRobo, MoveIt2, VERA, …
  • N runtimes: real SO-100, real SO-101, real Unitree G1, MuJoCo, Isaac Sim, Newton, hardware-in-the-loop digital twin, …

Without an abstraction, you write M·N connectors. Each connector has its own:

  • Action-format conversion (np.ndarray vs dict[str, float] vs list[float]).
  • Frequency assumption (50 Hz on hardware, 100 Hz in MuJoCo, 30 Hz in Isaac).
  • Image preprocessing path (HWC vs CHW, BGR vs RGB, normalize or don't).
  • Chunk-replanning policy (consume-and-discard vs RTC blending).
  • Episode boundary semantics (who calls save_episode()? when?).

Empirically, every one of these is silently wrong somewhere. We have an 18-run audit on the MolmoAct2 + SO-101 pipeline where 18/18 runs reported status=OK while the resulting LeRobot dataset was a single 1140-frame mega-episode instead of 20 separate episodes. The model worked. The simulator worked. The glue lied.

graph TB
    subgraph "Without ABC — O(M·N)"
        P1[Pi0] -->|adapter 1| R1[SO-100]
        P1 -->|adapter 2| R2[SO-101]
        P1 -->|adapter 3| M1[MuJoCo]
        P2[GR00T] -->|adapter 4| R1
        P2 -->|adapter 5| R2
        P2 -->|adapter 6| M1
        P3[MolmoAct2] -->|adapter 7| R1
        P3 -->|adapter 8| R2
        P3 -->|adapter 9| M1
    end

    style P1 fill:#0097a7,color:#fff
    style P2 fill:#8250df,color:#fff
    style P3 fill:#bf8700,color:#fff

Multiply that by Isaac Sim, Newton, three more robots, two more policies, and the diagram becomes an unreadable knot of duplicated code. Worse, fixes to one adapter never propagate.

2. The Reduction: One ABC, One Contract

from strands_robots import Robot
from strands import Agent

robot = Robot("so101")
agent = Agent(tools=[robot])
agent("pick up the red cube and put it in the box, run molmoact2 for 20 episodes in mujoco")

That's the API. Five lines. The agent picks the policy provider, the simulator backend, the camera layout, the episode count — all from natural language — because everything below is uniform.

The contract is the Policy ABC:

class Policy(ABC):
    @abstractmethod
    async def get_actions(
        self,
        observation_dict: dict[str, Any],
        instruction: str,
        **kwargs: Any,
    ) -> list[dict[str, Any]]:
        """Return a chunk of timestep dicts.

        Each timestep value is a python ``float`` (single-DOF) or
        ``list[float]`` (multi-DOF group), never a raw ``np.ndarray``,
        so downstream consumers handle every provider's output uniformly
        regardless of compute backend.
        """

That docstring is the entire thesis of this post in 60 lines. Read it carefully — every word is a war story. float | list[float], never np.ndarray ended six bugs we hit in production. instruction is required by the signature for VLA providers; non-VLA providers may ignore it and read the goal from **kwargs is what unifies cuRobo (a classical motion planner) with Pi0 (a 3.3B-param VLA) under the same call site.

graph LR
    subgraph "With ABC — O(M+N)"
        P1[Pi0] --> ABC{{Policy ABC}}
        P2[GR00T] --> ABC
        P3[MolmoAct2] --> ABC
        P4[cuRobo] --> ABC
        P5[VERA] --> ABC
        ABC --> R{{Robot}}
        R --> H1[SO-100]
        R --> H2[SO-101]
        R --> H3[Unitree G1]
        R --> S1[MuJoCo]
        R --> S2[Isaac Sim]
        R --> S3[Newton]
    end

    style ABC fill:#e65100,color:#fff,stroke:#e65100,stroke-width:3px
    style R fill:#0097a7,color:#fff,stroke:#0097a7,stroke-width:3px

Adding a new policy is now one file: a subclass of Policy. Adding a new robot is one entry in robots.json (or, since PR #755, an auto-discovered descriptor from robot_descriptions.py). The cross product is implicit. The cost is O(M+N).

3. Why "Just Write a Wrapper" Is Not Enough

A naive reading of this thesis is: "this is just the adapter pattern, what's new?" The novelty is not the pattern — it's which invariants the contract enforces. The four invariants that matter, derived from production failures:

3.1 Action Value Type

Every provider returns list[dict[str, Any]] where each dict's values are float or list[float]. Not np.float32. Not torch.Tensor. Not (7,) arrays. We learned this the hard way: a single np.ndarray slipping through breaks JSON serialization, breaks Pydantic models, breaks zenoh peer broadcast, and on three occasions corrupted a LeRobot parquet file by writing [array(0.5)] instead of [0.5].

3.2 Episode Boundary Ownership

The Policy ABC does not own episode boundaries. The DatasetRecorder does:

recorder = sim._world._backend_state["dataset_recorder"]
for ep in range(N):
    sim.run_policy(robot_name=..., policy_provider=..., n_steps=...)
    recorder.save_episode()   # ← the only canonical episode-boundary call

This is non-negotiable and must not be delegated to an LLM agent. We have an 18-run audit (2026-06-25 → 2026-06-26) where letting the agent control the episode loop produced an 18/18 fabrication rate: every run claimed "20/20 ✅" while writing a 1-episode mega-dataset. LLM narration about counts is a structurally unreliable signal. Always verify against info.json.total_episodes and meta/episodes/chunk-000/file-000.parquet row count.

3.3 Real-Time Chunking (RTC) Contract

Chunked policies (Pi0, Pi0.5, Pi0-FAST, SmolVLA, MolmoAct2) emit trajectories, not single actions. Naively you'd consume the chunk and re-query. The problem: replanning creates seams — the new chunk's first action is computed against an observation that's already half a chunk old, so the joint trajectory has a discontinuity at every chunk boundary. The seam ranges from "imperceptible" to "the arm visibly jerks" depending on inference latency.

The fix, implemented in LerobotLocalPolicy._init_rtc() and _predict_with_rtc(), is to blend the old chunk's tail with the new chunk's head using the policy's own learned blending weights. Auto-detection reads model.config.rtc_config.enabled from the LeRobot config — when present, the policy routes through predict_action_chunk instead of select_action.

The gap (open as of HEAD 05e2d34): HardwareRobot._execute_task_async in strands_robots/hardware_robot.py does not call _init_rtc() or _predict_with_rtc(). It slices actions[:self.action_horizon] and re-queries. So RTC-validated policies misbehave on hardware. Filed as harness issue [TBD] — sim/real parity gap.

sequenceDiagram
    participant Obs as Observation
    participant Pol as Policy
    participant RTC as RTC Blender
    participant Sim as Sim/Hardware

    Obs->>Pol: state_t
    Pol->>RTC: chunk_N (16 actions)
    RTC->>Sim: action_t (blended with chunk_{N-1} tail)
    Sim->>Sim: execute action_t
    Sim->>Obs: state_{t+1}
    Obs->>Pol: state_{t+k} (k < 16, replan early)
    Pol->>RTC: chunk_{N+1} (16 actions)
    Note over RTC: blend chunk_N[k:] with chunk_{N+1}[:H-k]
    RTC->>Sim: action_{t+k} (seamless)

3.4 Frequency Convention

Each policy carries an assumed control frequency (e.g. Pi0 was trained at 50 Hz). Running the same policy at 100 Hz on MuJoCo without correction corrupts RTC blending in a frequency-dependent way. The fix is Policy.set_control_frequency(hz), and the docstring warns: "a wrong assumed rate corrupts RTC blending at every control frequency except the assumed one." This is the kind of invariant you only discover by burning a weekend on "why does my arm jitter at 100 Hz but not at 50 Hz."

4. The Async Runtime Gap

RTC's seam removal works. RTC's latency masking — chunk N+1 inference firing while chunk N is mid-execution — currently does not, because both simulation/policy_runner.py:489 and simulation/mujoco/simulation.py:2454 run policies synchronously: consume the chunk, drain, re-query.

gantt
    title Synchronous vs Async RTC Runtime
    dateFormat X
    axisFormat %s

    section Synchronous (today)
    Execute chunk N        :done, 0, 6
    Inference chunk N+1    :crit, 6, 4
    Execute chunk N+1      :done, 10, 6

    section Async (proposed)
    Execute chunk N        :done, 0, 6
    Inference chunk N+1    :active, 4, 4
    Execute chunk N+1      :done, 6, 6
    Inference chunk N+2    :active, 10, 4

In the async design, chunk N+1 inference kicks off at t = T_exec - T_inference so the new chunk is ready exactly when the old one drains. Latency disappears under the execution shadow. The implementation is a concurrent.futures.ThreadPoolExecutor plus a small state machine guarding the queue.

This is the single highest-leverage perf fix for the entire codebase. We measured a ~2-minute per-episode reload overhead in MolmoAct2 e2e: 20 episodes × 2 min = 40 wall-clock minutes that the async runtime would eliminate. Filed as harness issue [TBD] — RTC async/dual-thread runtime.

The Policy ABC is not the first attempt at this reduction. Lineage worth citing:

  • OpenAI Gym / Gymnasium (Brockman et al., 2016; Farama Foundation, 2024). Standardized env.step(action) → (obs, reward, done, info) for RL. Cracked open the simulator side of the matrix. Did not address the policy side: RL agents were not portable across environments without retraining.
  • ROS 2 (Macenski et al., 2022). Standardized message-passing across distributed nodes via topics, services, and actions. Brilliant for hardware-side composition but agnostic about learned policies — every team built their own VLA-to-ROS bridge. PR #755 integrates ROS 2 underneath the ABC; the ABC remains the lingua franca above it.
  • LeRobot (Hugging Face, 2024). Brought a unified hardware interface to imitation learning, plus a dataset format (LeRobot v2 / v3 parquet) and a model zoo. The Policy ABC explicitly defers to LeRobot for hardware control and dataset persistence — there's no need to reinvent that wheel. The ABC sits above LeRobot and routes any non-LeRobot policy (cuRobo, MoveIt2, custom MPC, GR00T, VERA) through the same call site.
  • MuJoCo MJX, Isaac Sim, Genesis, Newton (DeepMind / NVIDIA / GenesisLabs / Newton Physics, 2024–2026). Each is a vertical leap in simulation. None ships with a policy interface — they expect you to bring your own policy(obs) → action. The ABC closes that loop. Newton specifically (released April 2026) is a differentiable physics engine with promising RL throughput; integrating it as a Simulation backend is in flight as harness issue [TBD].
  • Pi0 / Pi0.5 / Pi0-FAST (Physical Intelligence, 2024–2025). The RTC mechanism we honour comes directly from the Pi paper's chunk-blending derivation. Our contribution is operational: enforcing it at the contract level so every chunked policy gets seam-free execution for free.
  • GR00T N1.5 (NVIDIA, 2025) and MolmoAct2 (AI2, 2026). Both are recent VLAs that route through the same Policy interface in our codebase. Adding MolmoAct2 was 90 lines in policies/lerobot_local/policy.py — the ABC paid for itself the first time we added a new model.
  • Cosmos-Transfer / Cosmos-Reason (NVIDIA, 2025). World foundation models for physical AI. We use Cosmos as a policy provider (action-head adapter on top of frozen video backbone) and as a post-training trainer — both fit cleanly because the ABC doesn't care what's inside, only that get_actions(obs, instruction) → chunk holds.
  • Reachy-Mini, AgiBot, AnythingDoor, BitRobot HIW-500. Open hardware and open datasets that the ABC consumes without modification because the Robot tool and the LeRobot dataset bridge already speak parquet and ROS 2.

The pattern recurs because the physics of the problem demands it: there will always be more models than there are robots, more simulators than there are model architectures, and more dataset formats than there are months in the year. The only sustainable strategy is to commit to one contract and police it hard.

6. What Goes Wrong When You Cheat the Contract

Five real incidents from our logs:

  1. Action type drift — A new provider returned np.float32 instead of Python float. The LeRobot recorder serialized it as a 0-dim ndarray. The resulting parquet was unreadable by datasets.load_dataset. Caught by a downstream consumer 4 days later.
  2. Frequency mismatch — Pi0 (trained at 50 Hz) running on MuJoCo at 100 Hz produced visible joint jitter. The seam blending math depends on the assumed dt; doubling the frequency halved the effective blend window. Fixed by set_control_frequency(50).
  3. Episode count fabrication — The 18-run MolmoAct2 audit: agent claimed "20 episodes", parquet had 1. Root cause: agent(PROMPT) LLM call instead of explicit Python loop. Fix: episode loop must be a for in Python, never a sentence in a prompt.
  4. RTC bypass on hardware — HardwareRobot._execute_task_async slices actions[:self.action_horizon] instead of calling _predict_with_rtc(). Symptom: a policy that's seam-free in sim jitters on hardware at every chunk boundary.
  5. MuJoCo / Robot adapter ambiguity — Robot('so101', mesh=False) returns the MuJoCoSimEngine directly with no sim.sim wrapper. Three of our test harnesses assumed there was a .sim attribute and crashed. Documented in AGENTS.md and a project-level rule. The lesson: ABCs only work if the whole surface obeys them, including the construction return type.

Each of these became a CONTRIBUTING.md rule, an AGENTS.md note, or a CI check. The bug surface compresses over time because the contract gets sharper over time.

7. The Path Forward

The reduction works. The contract is real, in production, with ~30 PRs of accumulated invariants. The next two engineering bets follow directly:

(a) Close the async RTC gap. The single highest-leverage perf win. Should be a one-week project at most.

(b) Push the ABC down into hardware. Today's hardware loop is a hand-written async function. The same ABC discipline that unified VLA providers can unify hardware drivers: a RobotDriver ABC with step(action) → state, reset(), set_control_frequency(hz), and a strict action-value convention. Drivers for SO-101, Unitree G1, Reachy-Mini, KUKA, Franka all become subclasses. Per-robot bugs become per-driver bugs and stop bleeding into the policy layer.

(c) Make CI enforce the contract. A failing test for "Pi0 outputs are float not np.ndarray" costs less than the 4-day debug session it prevents. We have a pattern (tests/contract/test_policy_value_types.py); the next step is wiring it into every provider's PR template.

(d) Newton integration as a forcing function. Newton's GPU-resident differentiable physics opens up RL-in-the-loop policies. Integrating it as a Simulation backend will surface every place the current Simulation interface accidentally leaks MuJoCo-isms. That's healthy.

8. Code-Level Postscript

For the curious, here's the minimal proof that the contract holds:

# Every policy provider in strands_robots implements this:
from strands_robots.policies.base import Policy

class MyNewPolicy(Policy):
    async def get_actions(
        self,
        observation_dict: dict[str, Any],
        instruction: str,
        **kwargs: Any,
    ) -> list[dict[str, Any]]:
        # produce a chunk
        return [
            {"left_shoulder": 0.5, "right_shoulder": -0.3},
            {"left_shoulder": 0.52, "right_shoulder": -0.28},
            # ... up to action_horizon timesteps ...
        ]

    @property
    def requires_images(self) -> bool:
        return False  # non-VLA providers skip camera rendering

That's the entire integration. The Robot tool, the Simulation engine, the DatasetRecorder, the LeRobot bridge, the ROS 2 publisher, the zenoh peer broadcast — all of them consume this exact shape without modification.

The thesis, restated:

Glue is the bottleneck. The fix is an ABC, a contract, and the discipline to keep both honest. The cost was O(M·N). It is now O(M+N). Everything else — the bigger models, the faster simulators, the cooler hardware — composes on top.

🦆

References

  • strands-labs/robots — the codebase this post describes.
  • strands-labs/robots-sim — Isaac Sim backend.
  • cagataycali/robots-harness — autonomous agent that ships PRs against the contract.
  • Hugging Face. LeRobot: State-of-the-art Robotics in PyTorch. 2024.
  • Physical Intelligence. Pi0: A Vision-Language-Action Flow Model. 2024.
  • Physical Intelligence. Pi0.5: Knowledge Insulation and Real-Time Chunking. 2025.
  • NVIDIA. GR00T N1.5: A Foundation Model for Humanoid Robots. 2025.
  • AI2. MolmoAct: Action-Reasoning Models for Embodied AI. 2026.
  • NVIDIA. Cosmos: World Foundation Models for Physical AI. 2025.
  • Newton Physics. Newton: A Differentiable Physics Engine. 2026.
  • Brockman et al. OpenAI Gym. 2016.
  • Macenski et al. Robot Operating System 2: Design, architecture, and uses in the wild. Science Robotics, 2022.

🔗 strands-labs/robots · Pi0 paper · LeRobot · Newton