VLA Gallery / All models / Mobile manipulation

UMI on Legs

Stanford University·Columbia University·Google DeepMind · 2024-07

VAMobile manipulationNot discloseddiffusion

A diffusion manipulation policy trained on UMI handheld demonstrations outputs task-frame end-effector trajectories, which a manipulation-centric whole-body controller (WBC) trained with sim RL tracks to execute on a quadruped + arm.

Paper

UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body ControllersarXiv 2407.10353

Overview

Category

Mobile manipulation

Type

VA

Released

2024-07

Organization

Stanford University·Columbia University·Google DeepMind

Open weights

Partial

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

CLIP vision encoder (pretrained, variant not stated; wrist GoPro)

Action head

End-effector pose trajectory diffusion (position 3 + 6D rotation, up to 64 steps) → RL WBC (diffusion)

Chunk

64 steps (throwing) · 32 (pushing) · 16 (cup), executed 40 · 10 · 8

Control rate

Policy 5–12 Hz, WBC 50 Hz

Architecture (original summary)

Diffusion Policy (U-Net · DDIM, pretrained CLIP vision encoder) → end-effector pose trajectory in camera coordinates (3D position + 6D rotation) 64 steps (throwing) · 32 (pushing) · 16 (cup), executed at 5–12 Hz → transformed to task coordinates → RL MLP WBC outputs position targets for 12 leg · 6 arm joints at 50 Hz

Training

Pretraining accelerators

N/A

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Per-task training: WBC by sim RL (legged_gym PPO·IsaacGym, 4000 iter in sim comparison experiments), manipulation DP by per-task BC (throwing 500·pushing 25 trajectories; cup reuses public UMI checkpoint); GPU·time not disclosed

Data type

Human video / simulation·synthetic

Data scale

UMI handheld demonstrations: throwing 500·pushing 25 trajectories; cup rearrangement uses public checkpoint trained on 1,400 UMI trajectories; WBC trained in sim to track UMI end-effector trajectories

Main datasets

UMI demonstrations (throwing·pushing in-house collection; cup rearrangement from public UMI data)

Tasks & evaluation

Real robot, 20 trials per task: dynamic throwing 75% (mocap)/70% (iPhone), whole-body pushing (10 lb kettlebell) 95%/90%, cup rearrangement 85%/80% (UMI policy ported unmodified)

Robot

Unitree Go2 quadruped (12-DoF) + ARX5 6-DoF arm (Finray gripper·GoPro); localization via mocap or iPhone 15 Pro ARKit

Inference

Inference device & speed

DP: remote desktop RTX 4090 (over internet), runs at 5–12 Hz per task (designed as a 1–5 Hz low-frequency policy); WBC: MLP forward pass about 0.06 ms on Go2 onboard CPU, 50 Hz

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Remote RTX 4090 (policy) + Go2 onboard (WBC)

Reinforcement learning & base

Reinforcement learning

Partial (RL low-level controller) · Manipulation policy uses BC (diffusion) only; only the leg+arm WBC learns end-effector trajectory tracking with sim RL (legged_gym PPO)

RL evidence

https://arxiv.org/abs/2407.10353

Base control

Quadruped (Commands/latents → RL locomotion controller) · Policy outputs task-space end-effector trajectories → RL WBC generates leg and arm joint targets (base motion decided by WBC)

Notes

Key features

A diffusion manipulation policy trained on UMI handheld demonstrations outputs task-frame end-effector trajectories, which a manipulation-centric whole-body controller (WBC) trained with sim RL tracks to execute on a quadruped + arm. A public UMI policy for a fixed arm was ported to the quadruped without modification, and throwing·pushing·cup rearrangement all achieved 70%+ real-robot success.

Notes

WBC GPU·number of parallel environments·training time not stated in paper (README: based on legged_gym PPO·IsaacGym); CLIP encoder variant not stated (original UMI paper cup policy is ViT-B/16); code·WBC checkpoint·trajectory data released, release of throwing·pushing DP checkpoints unverified; CoRL 2024. Based on: Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots (arXiv 2402.10329) — extends UMI demonstration data·policy·code to a quadruped platform (stated in paper·README)

Verification

Confirmed