VLA GalleryAll modelsGallery

VLA Gallery / All models / Industry models

RT-H

Google DeepMind·Stanford University · 2024-03

VLAIndustry models55Bdiscrete tokens

Hierarchical VLA in which one model first predicts a language motion such as 'move arm forward', then predicts actions.

View architecture diagram in the galleryPaper ↗

Paper

RT-H: Action Hierarchies Using LanguagearXiv 2403.01823

Overview

Category
Industry models
Type
VLA
Released
2024-03
Organization
Google DeepMind·Stanford University
Open weights
No
Parameters
55B (based on PaLI-X 55B)

Architecture

Backbone LLM
UL2 32B · PaLI-X 55B config
Vision encoder
ViT (PaLI-X)
Action head
language motion → action tokens (discrete tokens)
Chunk
Not disclosed
Control rate
Not disclosed
Architecture (original summary)
Single PaLI-X 55B model (ViT encoder + encoder-decoder); language motion query then action query; action space is end-effector delta position·rotation, gripper, base movement

Training

Pretraining accelerators
Not disclosed
Pretraining duration
Not disclosed
Pretraining steps & batch
batch 1024, lr 1e-3 (co-training with PaLI-X training mixture; steps not disclosed)
GPU/TPU-hours
Not disclosed
Fine-tuning
Not disclosed (intervention training uses 30 correction episodes per task)
Data type
Real-robot demonstrations + web vision-language (PaLI-X training mixture)
Data scale
Kitchen 70K demos (6 semantic task categories); Diverse+Kitchen 100K demos (24 categories); 2,500+ auto-labeled language motions
Main datasets
In-house Kitchen and Diverse demonstration data, PaLI-X training mixture
Tasks & evaluation
Average +15% over RT-2 across 8 tasks; 40%→63% with intervention learning; 50% above RT-2-IWR; new objects 65% (RT-2 55%)
Robot
Mobile manipulator (platform name Unverified)

Inference

Inference device & speed
Latency mitigated by asynchronous querying; hardware·latency·control frequency not disclosed
Inference memory
Not disclosed
Weights bf16 (calc.)
110 GB
Where it runs
Not disclosed

Reinforcement learning & base

Reinforcement learning
Not used · Imitation learning + human language motion correction data
Base control
Wheeled (Direct base velocity/pose output) · Directly outputs 2 base-motion dims out of 9 policy action dims (language motions such as 'turn base left')

Notes

Key features
Hierarchical VLA in which one model first predicts a language motion such as 'move arm forward', then predicts actions. Humans intervene at the language-motion level to correct it, and it retrains on that correction data.
Notes
No TPU·GPU statement in paper. arXiv v1 2024-03-04. Affiliations are Google DeepMind and Stanford University.
Verification
Confirmed

Other models in this category