VLA Gallery / All models / Humanoid whole-body controllers

Humanoid Locomotion as Next Token Prediction

UC Berkeley · 2024-02

VAHumanoid whole-body controllers1M / 2M / 8Mregression

Treats real-world locomotion as next-token prediction and jointly trains on trajectories from RL policies·model-based controllers·motion capture·YouTube videos.

Paper

Humanoid Locomotion as Next Token PredictionarXiv 2402.19469

Overview

Category

Humanoid whole-body controllers

Type

VA

Released

2024-02

Organization

UC Berkeley

Open weights

Unverified

Parameters

1M / 2M / 8M (embedding 144/192/384, heads 3/4/12, blocks 4/4/6)

Architecture

Backbone LLM

None

Vision encoder

—

Action head

Next-token prediction (observations·actions) (regression)

Chunk

Not disclosed

Control rate

Not disclosed

Architecture (original summary)

Autoregressive transformer (default hidden 192, 4 layers), sequence of observation and action tokens, context 16 (32 and 48 also tested); next-token prediction within the same modality, missing actions filled with mask token [M] and excluded from the loss

Training

Pretraining accelerators

Not disclosed

Pretraining duration

Not disclosed

Pretraining steps & batch

Not disclosed

GPU/TPU-hours

Not disclosed

Fine-tuning

N/A

Data type

Simulation·synthetic / motion capture / human video

Data scale

10k neural-network policy trajectories (10 s each, flat ground), 2 sets of 10k model-based controller trajectories (10 s each), ~1k of ~4k KIT motion capture clips, YouTube videos (PHALP-processed, count not disclosed)

Main datasets

Agility Robotics simulator trajectories, KIT motion capture, YouTube

Tasks & evaluation

Zero-shot walking for one week at multiple locations in San Francisco; distance·success counts not disclosed

Robot

Agility Digit

Inference

Inference device & speed

Control rate not disclosed

Inference memory

Not disclosed

Weights bf16 (calc.)

0 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Partial (RL-generated data) · Next-token prediction (supervised learning). Part of the training data is 10k simulation trajectories from a large-scale RL policy

RL evidence

https://arxiv.org/html/2402.19469

Base control

Humanoid legged (Direct whole-body joint/pose output) · Transformer autoregressively outputs joint position actions for omnidirectional velocity-command-tracking locomotion (Digit, real walking in San Francisco)

Notes

Key features

Treats real-world locomotion as next-token prediction and jointly trains on trajectories from RL policies·model-based controllers·motion capture·YouTube videos. Transfers to the real world with only 27 h of locomotion data.

Notes

Parameters (B) is the largest variant, 8M. '27 hours of walking data' describes a training condition and there is no total data-hours figure, so data hours is null. All training compute not disclosed (confirmed in verification).

Verification

Confirmed