VLA GalleryAll modelsGallery

VLA Gallery / All models / Reference baselines (pre-2024)

RT-2-X

Open X-Embodiment Collaboration (21 institutions) · 2023-10

VLAReference baselines (pre-2024)55Bdiscrete tokens

Co-fine-tunes RT-2 on original VLM data and the OXE robot mix at about 1:1.

View architecture diagram in the galleryPaper ↗

Paper

Open X-Embodiment: Robotic Learning Datasets and RT-X ModelsarXiv 2310.08864

Overview

Category
Reference baselines (pre-2024)
Type
VLA
Released
2023-10
Organization
Open X-Embodiment Collaboration (21 institutions)
Open weights
No
Parameters
55B (OXE paper Table II also has a 5B variant)

Architecture

Backbone LLM
UL2 32B encoder-decoder (PaLI-X 55B)
Vision encoder
ViT (PaLI-X)
Action head
256-bin text tokens (discrete tokens)
Chunk
None (1 step)
Control rate
3–10 Hz (per robot)
Architecture (original summary)
ViT + UL2 VLM backbone (RT-2 style); 8-dim (1 termination·7 end-effector motion) 256-bin text tokens

Training

Pretraining accelerators
Not disclosed (uses an existing VLM)
Pretraining duration
Not disclosed
Pretraining steps & batch
Not disclosed
GPU/TPU-hours
Not disclosed
Fine-tuning
Not disclosed; co-fine-tuning with original VLM data and robot data at about 1:1
Data type
Real-robot teleoperation etc. (collection method varies by contributed dataset); web vision-language
Data scale
OXE training mix (9 embodiments) + original VLM data, about 1:1; full OXE 1M+ trajectories·22 embodiments
Main datasets
OXE subset: RT-1, QT-Opt, Bridge, Task Agnostic Robot Play, Jaco Play, Cable Routing, RoboTurk, NYU VINN, Austin VIOLA, Berkeley Autolab UR5, TOTO, Language Table; original VLM training data
Tasks & evaluation
3,600 evaluations on 6 robot types (jointly with RT-1-X); emergent skills about 3× RT-2
Robot
Training mixture of 9 manipulators (out of 22 embodiments in all of OXE); 6 evaluation robots

Inference

Inference device & speed
Hosted as a cloud service with network queries, 3–10 Hz per robot
Inference memory
Not disclosed
Weights bf16 (calc.)
110 GB
Where it runs
Cloud

Reinforcement learning & base

Reinforcement learning
Not used · VLM·robot data co-fine-tuning (supervised learning)
Base control
Fixed · No base control (8-dim actions: terminate 1 + EE movement 7; Google mobile robot is arm only too)

Notes

Key features
Co-fine-tunes RT-2 on original VLM data and the OXE robot mix at about 1:1. Achieved about 3× RT-2's performance on emergent skill evaluations.
Notes
No RT-2-X checkpoint in README. The 5B variant is a new fact from verification notes. Institution attribution corrected. Parameters (B) is 55B.
Verification
Corrections applied

Other models in this category