VLA Gallery / All models / Reference baselines (pre-2024)

RT-2

Google DeepMind · 2023-07

VLAReference baselines (pre-2024)55B · 5Bdiscrete tokens

Co-fine-tunes a pretrained VLM on robot data and web VQA·caption data, outputting actions as text tokens.

Paper

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlarXiv 2307.15818

Overview

Category

Reference baselines (pre-2024)

Type

VLA

Released

2023-07

Organization

Google DeepMind

Open weights

No

Parameters

55B · 5B (RT-2-PaLI-X), 12B (RT-2-PaLM-E); 3B for Language-Table (PaLI-3B: ViT-G/14 2B + UL2-3B)

Architecture

Backbone LLM

UL2 32B encoder-decoder (PaLI-X 55B) / PaLM family (PaLM-E 12B)

Vision encoder

ViT-22B (PaLI-X) / ViT-4B (PaLM-E)

Action head

8 text tokens of 256 bins (discrete tokens)

Chunk

None (1 step)

Control rate

1–3 Hz (55B)

Architecture (original summary)

PaLI-X (ViT-22B + 32B·50-layer encoder-decoder) or PaLM-E (ViT-4B + 12B decoder); outputs 8 action integers (6-DoF displacement·gripper·terminate) as 256-bin text tokens, no chunks

Training

Pretraining accelerators

Not disclosed (uses existing PaLI-X·PaLM-E pretrained weights)

Pretraining duration

Not disclosed

Pretraining steps & batch

Not disclosed (existing VLM)

GPU/TPU-hours

Not disclosed

Fine-tuning

Accelerator·time not disclosed; co-fine-tuning 55B: lr 1e-3·batch 2048·80K steps, 5B: batch 2048·270K steps, 12B: lr 4e-4·batch 512·1M steps, 3B (Language-Table): batch 128·300K steps

Data type

Real-robot teleoperation; web vision-language

Data scale

Robot: RT-1 data (13 robots·17 months, about 130k episodes per the RT-1 paper); web: WebLI about 10B image-text pairs (109 languages) filtered to the top 10% for 1B examples + captioning·VQA datasets; robot share about 50% (PaLI-X)·about 66% (PaLM-E)

Main datasets

RT-1 data; WebLI; captioning·VQA datasets

Tasks & evaluation

6k real-robot evaluation trials; 200+ training tasks, 280+ generalization tasks; Language-Table simulation; emergent capability evaluation

Robot

Same mobile manipulator as RT-1 (office kitchen)

Inference

Inference device & speed

Queried over the network on a multi-TPU cloud service; 55B 1–3 Hz, 5B about 5 Hz; states it cannot run directly on the robot's onboard GPU

Inference memory

Not disclosed

Weights bf16 (calc.)

110 GB

Where it runs

Cloud

Reinforcement learning & base

Reinforcement learning

Not used · Robot·web data co-fine-tuning (supervised learning)

Base control

Fixed · No base control (a mobile manipulator, but actions are only EE 6-DoF displacement·gripper·terminate)

Notes

Key features

Co-fine-tunes a pretrained VLM on robot data and web VQA·caption data, outputting actions as text tokens. Its 256-bin action token scheme was reused by many later autoregressive VLAs.

Notes

Parameters (B) is the largest variant, 55B. Training compute disclosed only as steps × batch. The original does not give a vision/language breakdown of PaLI-X-5B.

Verification

Confirmed