WholeBodyVLA

Fudan University·OpenDriveLab & MMLab(HKU)·AgiBot·SII · 2025-12

VLAHumanoid VLAPrismatic-7B-basedlatent actions

Large-space loco-manipulation through unified latent learning from low-cost egocentric video without action labels.

Paper

Backbone LLM: Llama 2 7B · Prismatic-7B config
Vision encoder: DINOv2 LAM + Prismatic vision
Action head: Latent action decoder (manipulation, locomotion) (latent actions)
Chunk: Not disclosed
Control rate: Not disclosed
Architecture (original summary): Prismatic-7B VLA + DINOv2-based VQ-VAE latent action models (separate manipulation·locomotion LAMs); latent actions decoded at about 10 Hz into bimanual joint actions and locomotion commands, locomotion commands executed by the LMO RL policy at 50 Hz

Pretraining accelerators: H100 × 8 (LAM·VLA); LMO RL on H100 × 1
Pretraining duration: Not disclosed
Pretraining steps & batch: VLA 20,000 steps · batch 1,024; LAM 30,000 steps · batch 256
GPU/TPU-hours: Not disclosed
Fine-tuning: 10,000 steps · batch 64 (GPU within the H100 × 8 sentence scope), time not disclosed
Data type: Human video / real-robot teleoperation
Data scale: About 300 hours of egocentric walking video; AgiBot World manipulation data (scale not disclosed); 50 teleoperation runs per task
Main datasets: In-house egocentric walking video, AgiBot World
Tasks & evaluation: Bag Packing · Box Loading · Cart Pushing average 78.0%; +21.3% over previous baselines
Robot: AgiBot X2 (7-DoF arms + Omnipicker gripper, 6-DoF legs, 1-DoF waist)

Inference device & speed: VLA on RTX 4090 workstation (about 10 Hz), LMO RL on onboard NanoPi (50 Hz)
Inference memory: Not disclosed
Weights bf16 (calc.): Not disclosed
Where it runs: Local PC (VLA) + onboard (LMO)

Reinforcement learning: Partial (RL low-level controller) · VLA by supervised learning of latent actions; locomotion commands executed by LMO RL policy (trained on 1 H100)
Base control: Humanoid legged (Commands/latents → RL locomotion controller) · VLA outputs upper-body joint angles + locomotion commands (3-axis discrete directives for forward/back·left/right·turn + posture height) → LMO RL policy outputs lower-body torques (50 Hz)

Key features: Large-space loco-manipulation through unified latent learning from low-cost egocentric video without action labels. 21.3% higher than prior baselines.
Notes: Fine-tuning settings and deployment hardware corrected from the notes' 'Not disclosed' in verification; LAM settings are NEW. Data hours of 300 are walking video only. Parameters (B) is null because total parameters are Not disclosed. LMO simulator·env·time Not disclosed.
Verification: Corrections applied