VLA GalleryAll modelsGallery

VLA Gallery / All models / VA

UMI

Stanford University; Columbia University; Toyota Research Institute · 2024-02

VAVANot discloseddiffusion

Collects demonstrations without a robot using a handheld gripper with a GoPro and transfers them to robot policies.

View architecture diagram in the galleryPaper ↗

Paper

Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild RobotsarXiv 2402.10329

Overview

Category
VA
Type
VA
Released
2024-02
Organization
Stanford University; Columbia University; Toyota Research Institute
Open weights
Partial
Parameters
Not disclosed

Architecture

Backbone LLM
None
Vision encoder
CLIP ViT-B/16 (ResNet-34 · ViT-L/14 per task)
Action head
CNN U-Net diffusion (DDIM 16 steps) (diffusion)
Chunk
Executes 6 steps
Control rate
10 Hz (throwing 20 Hz)
Architecture (original summary)
Diffusion Policy CNN U-Net (DiffusionUnetTimmPolicy, down_dims 256·512·1024) + vision encoder CLIP ViT-B/16 (cup · dishwashing) · ResNet-34 (throwing · cloth, no pretraining) · CLIP ViT-L/14 (outdoor); DDIM 50 training · 16 inference steps, action Ta 6, 10 Hz (throwing 20 Hz)

Training

Pretraining accelerators
N/A
Pretraining duration
N/A
Pretraining steps & batch
N/A
GPU/TPU-hours
Not disclosed
Fine-tuning
Per task (GPU·epochs·batch): cup 4× A10G·250·512, tossing 4× A10G·350·1024, cloth 4× A10G·100·1024, dishwashing 8× A10G·90·224, in-the-wild cup 8× A100·50·512; time Not disclosed; README 'training verified on a single RTX3090 24GB GPU'
Data type
Human video (handheld GoPro gripper demonstrations, SLAM tracking)
Data scale
Per task: cup 305, throwing 280, cloth 250, dishwashing 258 episodes; outdoor cup 1,400 demos (30 locations, 12 person-hours, 3 demonstrators)
Main datasets
UMI in-house collection
Tasks & evaluation
Cup arrangement, dynamic throwing, bimanual cloth folding, dishwashing, outdoor cup arrangement (30 locations)
Robot
Single-arm·bimanual robots (model Not stated in notes); collection via handheld gripper

Inference

Inference device & speed
Hardware and latency not disclosed
Inference memory
Not disclosed
Weights bf16 (calc.)
Not disclosed
Where it runs
Not disclosed

Reinforcement learning & base

Reinforcement learning
Not used · Imitation learning from human handheld demonstrations
Base control
Fixed · No base control (arm only; data collected with handheld gripper)

Notes

Key features
Collects demonstrations without a robot using a handheld gripper with a GoPro and transfers them to robot policies. Matches sensor and robot latencies; 3 demonstrators collected 1,400 demos in 12 person-hours at 30 locations.
Notes
Gripper BoM $73, GoPro·accessories $298; 155° fisheye lens·side mirrors·IMU-aided ORB-SLAM3. In-the-wild cup checkpoint (fine-tuned CLIP ViT-L) released. Repository default config (120 epochs, batch 64) differs from the paper's per-task settings.
Verification
Confirmed

Other models in this category