Robot Utility Models (RUM)

New York University; Hello Robot Inc.; Meta · 2024-09

VAVANot discloseddiscrete tokens

Uses a $25-BoM iPhone collection tool to gather about 1,000 demos per task across about 40 environments, deploying zero-shot in new environments.

Paper

Backbone LLM: None
Vision encoder: ResNet34 (HPR init)
Action head: VQ action codes (discrete tokens)
Chunk: Not disclosed
Control rate: Not disclosed
Architecture (original summary): HPR-initialized ResNet34 vision encoder + VQ-BeT policy (chosen over DP for performance·robot CPU latency); iPhone camera input

Pretraining accelerators: N/A
Pretraining duration: N/A
Pretraining steps & batch: N/A
GPU/TPU-hours: Not disclosed
Fine-tuning: Per-task models: 2 A100, 24–48 hours (phrased as 'all of our models', so unclear whether per model), 500 epochs
Data type: Human video (handheld iPhone Stick-v2 demonstrations)
Data scale: About 1,000 demos per task, about 40 environments (about 25 per environment on average); door opening 1,200, drawer 525
Main datasets: Stick-v2 in-house collection
Tasks & evaluation: Opening cabinet doors, opening drawers, picking up tissues, picking up paper bags, righting fallen objects; zero-shot deployment in unseen environments
Robot: Hello Robot Stretch (also tested on xArm 7)

Inference device & speed: Stretch onboard CPU, iPhone camera; latency not disclosed
Inference memory: Not disclosed
Weights bf16 (calc.): Not disclosed
Where it runs: Onboard (Stretch built-in CPU)

Reinforcement learning: Not used · Imitation learning. At deployment, a gpt-4o verifier detects failures and retries
Base control: Fixed · Navigation assumed to be a separate component; policy outputs only EE-relative 6D actions at the task location

Key features: Uses a $25-BoM iPhone collection tool to gather about 1,000 demos per task across about 40 environments, deploying zero-shot in new environments. An external GPT-4o verifier detects failures and triggers retries.
Notes: The verifier gpt-4o-2024-05-13 is a component external to the policy; the policy itself takes no language input. Paper states release of code·data·models·hardware design.
Verification: Confirmed