EgoMimic

Georgia Institute of Technology; Stanford University · 2024-10

VAVANot disclosedregression

Co-trains one policy on first-person human demos captured with Project Aria glasses and robot teleoperation.

Paper

Backbone LLM: None
Vision encoder: ResNet18†
Action head: SE(3) pose + joint decoder (regression)
Chunk: 100 steps†
Control rate: 25 Hz (inference 1 Hz)
Architecture (original summary): ACT-based shared Transformer (4 encoder · 7 decoder layers, hidden 512, FF 3200, 8 heads; ResNet18†) + SE(3) pose decoder shared by human and robot + robot joint decoder; chunk 100†

Pretraining accelerators: N/A (human-robot co-training)
Pretraining duration: N/A
Pretraining steps & batch: N/A
GPU/TPU-hours: Not disclosed
Fine-tuning: Per task: 4× A40, 120,000 iterations, global batch 128, about 24 hours ('All models'); lr 5e-5
Data type: Human video; real-robot teleoperation
Data scale: Human: Object-in-Bowl 1,400 (60 min), Groceries 160 (80 min), Laundry 590 (100 min); robot: 270 (120 min), 300 (300 min), 430 (300 min)
Main datasets: Self-collected human data with Project Aria; in-house robot teleoperation
Tasks & evaluation: Continuous Object-in-Bowl, Groceries, Laundry
Robot: ViperX 300 S bimanual + Project Aria glasses (human data)

Inference device & speed: Inference 1 Hz, control 25 Hz on an RTX 4090 desktop
Inference memory: Not disclosed
Weights bf16 (calc.): Not disclosed
Where it runs: Local PC

Reinforcement learning: Not used · Joint imitation learning on human·robot data
Base control: Fixed · No base control (ViperX fixed bimanual)

Key features: Co-trains one policy on first-person human demos captured with Project Aria glasses and robot teleoperation. On Object-in-Bowl, human data yields 1,400 demos in 60 minutes vs 270 robot demos in 120 minutes.
Notes: ResNet18·chunk 100 not found in verification extraction, so unverifiable (†). Collection time disclosed only in minutes, so the hours (h) cell is left blank.
Verification: Confirmed