VLA GalleryAll modelsGallery

VLA Gallery / All models / Autoregressive & latent-action VLA

3D-VLA

UMass Amherst·SJTU·SCUT·Wuhan University·MIT·UCLA·MIT-IBM Watson AI Lab · 2024-03

VLAAutoregressive & latent-action VLANot discloseddiscrete tokens

Combines 3D scene understanding and goal state generation (diffusion world model) with action token prediction.

View architecture diagram in the galleryPaper ↗

Paper

3D-VLA: A 3D Vision-Language-Action Generative World ModelarXiv 2403.09631

Overview

Category
Autoregressive & latent-action VLA
Type
VLA
Released
2024-03
Organization
UMass Amherst·SJTU·SCUT·Wuhan University·MIT·UCLA·MIT-IBM Watson AI Lab
Open weights
Partial
Parameters
Not disclosed (BLIP2-FlanT5-XL based)

Architecture

Backbone LLM
FlanT5-XL · BLIP2 config
Vision encoder
3D features (3D-LLM design)
Action head
Discrete action tokens + goal image·point cloud diffusion (discrete tokens)
Chunk
Not disclosed
Control rate
Not disclosed
Architecture (original summary)
3D-LLM design, BLIP2-FlanT5-XL; discrete action tokens (<aloc0-255>·<arct0-255>·<gripper0/1>) + goal image · depth · point cloud diffusion models

Training

Pretraining accelerators
LLM: '6 × 32 V100s' (alignment stage '6 × 64 V100s'); goal LDM: 192/96 V100; Point-E: 32 V100
Pretraining duration
LLM not disclosed; RGB goal LDM 3 weeks, RGB-D LDM 2 weeks
Pretraining steps & batch
LLM 30 epochs, batch 4 per node (alignment stage up to 20 epochs, batch 2); LDM 360k/100k steps; Point-E 200 epochs
GPU/TPU-hours
Not disclosed
Fine-tuning
Not disclosed
Data type
Real-robot teleoperation + simulation·synthetic + human video
Data scale
2M 3D-language-action pairs; 316k episodes
Main datasets
12 OXE datasets, RLBench, CALVIN, Dobb-E, RH20T, Epic-Kitchens, HOI4D
Tasks & evaluation
RLBench (Put Knife 68%, Take Umbrella 52%, Pick Cup 40%); CALVIN (single task 44.7%, 2 in a row 16.3%)
Robot
RLBench·CALVIN simulation

Inference

Inference device & speed
Not disclosed
Inference memory
Not disclosed
Weights bf16 (calc.)
Not disclosed
Where it runs
Not disclosed

Reinforcement learning & base

Reinforcement learning
Not used · Supervised learning
Base control
Fixed · No base control (RLBench·CALVIN arm sim)

Notes

Key features
Combines 3D scene understanding and goal state generation (diffusion world model) with action token prediction. Built 2M 3D-language-action pairs from robot data and human-object interaction videos.
Notes
'6 × 32 V100s' is verbatim and ambiguous between 6 nodes × 32 and 192 in total. The goal LDM (192 V100 · 3 weeks) is a diffusion model separate from the policy, so it is not counted in the GPU-hours field. Only the diffusion goal model is released on HF.
Verification
Confirmed

Other models in this category