VLA GalleryAll modelsGallery

VLA Gallery / All models / Autoregressive & latent-action VLA

Octo

UC Berkeley·Stanford·CMU·Google DeepMind · 2023-12

VAAutoregressive & latent-action VLA27Mdiffusion

A generalist VA policy trained on OXE 800k episodes without a pretrained VLM.

View architecture diagram in the galleryPaper ↗

Paper

Octo: An Open-Source Generalist Robot PolicyarXiv 2405.12213

Overview

Category
Autoregressive & latent-action VLA
Type
VA
Released
2023-12
Organization
UC Berkeley·Stanford·CMU·Google DeepMind
Open weights
Yes
Parameters
27M (Small)/93M (Base), excluding T5-base 111M; 2023-12 technical report Table 4 gives Base 86M

Architecture

Backbone LLM
None (T5-base text encoder)
Vision encoder
ViT patch tokens
Action head
Diffusion head (3-layer MLP, 20 steps) (diffusion)
Chunk
4 steps (fine-tuning default)
Control rate
5 · 10 · 15 Hz (per robot)
Architecture (original summary)
ViT-S/B-sized Transformer + T5-base language encoder (111M); diffusion action head (3-layer MLP, 20 steps), action chunk (fine-tuning default 4), 2-frame observation history

Training

Pretraining accelerators
TPU v4-128 pod
Pretraining duration
Base 14 h (paper); Small 8 h (README)
Pretraining steps & batch
300k steps, batch 2048(ViT-B)
GPU/TPU-hours
1,792 (Converted · TPU v4 core-hours (128 × 14 h, Octo-Base))
Fine-tuning
1 × A5000 (24GB) about 5 h (50k steps, about 100 target demos); repo default batch 256, head_only · head_mlp_only · full modes
Data type
Real-robot teleoperation
Data scale
800k episodes (25 OXE datasets); preprocessed data about 1.2TB (README)
Main datasets
Open X-Embodiment (Fractal·Kuka·Bridge each weighted 17.0%)
Tasks & evaluation
9 real-robot setups at 4 institutions (WidowX BridgeV2, UR5 TableTop, RT-1 Robot, Berkeley Bimanual, etc.); LIBERO average 75.1% (secondary, OpenVLA README)
Robot
9 setups including WidowX, UR5, RT-1 robot, bimanual setup; evaluation control rates 5·10·15 Hz

Inference

Inference device & speed
Single RTX 4090: Base 13 it/s, Small 17 it/s (README); no inference speed in paper
Inference memory
Not disclosed
Weights bf16 (calc.)
0.2 GB
Where it runs
Local PC

Reinforcement learning & base

Reinforcement learning
Not used · Imitation learning on OXE data
Base control
Fixed · No base control (arm EE actions; RT-1 robot is arm only)

Notes

Key features
A generalist VA policy trained on OXE 800k episodes without a pretrained VLM. Modular attention allows fine-tuning to new sensors and action spaces.
Notes
Release month is based on the repository v0.1.0 tag (2023-12-14) and the technical report; arXiv v1 is 2024-05-20. Parameters (B) refer to Octo-Base (excluding T5); GPU·TPU-hours were converted treating the 128 in TPU v4-128 as the core count.
Verification
Confirmed

Other models in this category