Humanoid-VLA

Westlake University·Westlake Robotics·Zhejiang University · 2025-02

VLAHumanoid VLALlama3-70B-baseddiscrete tokens

A hierarchical structure that generates whole-body motion from language and egocentric vision, executed by an RL tracker.

Paper

Backbone LLM: Llama 3 70B
Vision encoder: —
Action head: Per-body-part VQ motion tokens (5 parts) (discrete tokens)
Chunk: Not disclosed
Control rate: Not disclosed
Architecture (original summary): Llama3-70B-based; hierarchical VQ motion tokenizer (5 body-part tokens: left·right legs, torso, left·right arms, codebook 1,024); vision via cross-attention with language tokens as queries; generated motion converted to joint commands by an IsaacGym PPO goal-conditioned RL policy

Pretraining accelerators: H100 × 8
Pretraining duration: 216 hours
Pretraining steps & batch: Steps not disclosed · per-device batch 4 (lr 2e-5, cosine, warmup 0.01)
GPU/TPU-hours: 1,728 (Converted · H100-hours (8 × 216 h))
Fine-tuning: Not disclosed; low-level RL tracker uses IsaacGym (env count·GPU·time Not disclosed)
Data type: Motion capture / human video / simulation·synthetic
Data scale: Motion capture 29K clips·0.3M frames·4.1 h; online video 0.8M clips·541M frames·7515.7 h; synthetic 100K clips·16M frames·227.7 h; Total 0.929M clips·557.3M frames·7790.2 h
Main datasets: Built in-house (motion capture, online video, synthetic motion); evaluation on HumanML3D, Humanoid-S
Tasks & evaluation: Real Unitree G1 8 tasks × 10 trials (ball kick 9/10; obstacle avoidance·turning toward object·object grasping·hand waving·jumping·dancing·punching); HumanML3D FID 0.467±.018, DIV 4.585±.086; new Humanoid-S benchmark (4,646 manually annotated videos)
Robot: Unitree G1 whole body

Reinforcement learning: Partial (RL low-level controller) · LLM motion token generation is supervised; execution by an IsaacGym PPO goal-conditioned RL tracker
Base control: Humanoid legged (Commands/latents → RL locomotion controller) · VLA generates per-part motion tokens including legs → goal-conditioned RL whole-body controller converts them to PD joint targets (obstacle-avoiding movement·jumping·kicking)

Key features: A hierarchical structure that generates whole-body motion from language and egocentric vision, executed by an RL tracker. <Track>·<Time>·<Occlusion>·<State> self-supervised augmentation produces 25× more data than prior work without manual labels.
Notes: Total row was missing from notes but verification confirmed it exists in both HTML·PDF (NEW). Source Total hours 7790.2 h does not match the sum of the three rows' hours (clip·frame sums match); copied as in source. Parameters (B) null since total parameters not disclosed (base LLM Llama3-70B).
Verification: Confirmed