HAT

HAT (Humanoid Policy ~ Human Policy, PH²D)

UC San Diego·CMU·University of Washington·MIT·Apple · 2025-03

VAHumanoid VLANot disclosedCVAE

Co-trains human VR demonstrations and humanoid teleoperation in the same 54-D action space.

Paper

Backbone LLM: None
Vision encoder: DINOv2 ViT-S (frozen)
Action head: Human-centric 54-D action chunks (CVAE)
Chunk: Not disclosed
Control rate: Not disclosed
Architecture (original summary): ACT-family Human Action Transformer; frozen DINOv2 ViT-S + mid-layer visual adapter (replacing ACT's ResNet-18); 54-dim human-centric unified action space (head · both wrists rotation 6D, wrist position xyz, 3D keypoints of 10 fingertips), action chunks

Pretraining accelerators: Not disclosed
Pretraining duration: Not disclosed
Pretraining steps & batch: Not disclosed
GPU/TPU-hours: Not disclosed
Fine-tuning: N/A
Data type: Human video / real-robot teleoperation
Data scale: 26,824 human demonstrations (~3.02M frames), 1,552 robot demonstrations (~668k frames); frame counts estimated by the source assuming 30 Hz
Main datasets: PH²D (Apple Vision Pro, Meta Quest 3·Apple Vision Pro + ZED camera)
Tasks & evaluation: OOD evaluation: cup handover 52/60, horizontal grasp 12/30, vertical grasp 29/70, pouring 8/10
Robot: Unitree H1 + 6-DoF Inspire hands (bimanual manipulation)

Reinforcement learning: Not used · Imitation learning on human·robot data
Base control: Fixed · No base control (lower body explicitly not actuated; head·wrist·fingertip 54-dim only)

Key features: Co-trains human VR demonstrations and humanoid teleoperation in the same 54-D action space. Human grasp demonstrations are collected in 3.79±0.27 s, faster than 19.72±1.65 s for teleoperation.
Notes: No language conditioning, so type VA. README training example (NEW; a MuJoCo sim example, not the paper setup): single GPU (24GB+ VRAM), chunk 100, batch 64, 50000 epochs, lr 1e-4.
Verification: Confirmed