VLA Gallery / All models / Autoregressive & latent-action VLA

CoT-VLA

Stanford·NVIDIA·MIT · 2025-03

VLAAutoregressive & latent-action VLA7Bdiscrete tokens

Visual chain-of-thought (visual CoT) VLA that first generates a subgoal image.

Paper

Backbone LLM: VILA-U 7B (based on Llama 2 7B)
Vision encoder: VILA-U vision tokenizer
Action head: Subgoal image generation → 256-bin tokens (discrete tokens)
Chunk: 10 steps
Control rate: Not disclosed
Architecture (original summary): VILA-U 7B (unified understanding·generation); generates subgoal image then chunk of 10 actions, 7 256-bin tokens per action (full attention over actions)

Pretraining accelerators: 96 × A100 (8 × 12 nodes)
Pretraining duration: Not disclosed (only the total 11K A100-hours stated)
Pretraining steps & batch: 10 epochs, global batch 2048, lr 1e-4, 256×256 images
GPU/TPU-hours: 11,000 (Stated · A100-hours)
Fine-tuning: 1 A100 node, 10–24 h (depending on data size), lr 1e-5 · 150 epochs
Data type: Real-robot teleoperation + human video
Data scale: Not disclosed (mixture ratio only: Bridge 24.14%, SSv2 3.45%, EPIC-KITCHENS-100 3.45%)
Main datasets: OXE subset, Something-Something V2, EPIC-KITCHENS-100
Tasks & evaluation: LIBERO average 81.13%; Bridge-V2 WidowX; Franka-Tabletop
Robot: WidowX, Franka, LIBERO simulation

Inference device & speed: Absolute speed not disclosed; on average 7× slower at chunk 10
Inference memory: Not disclosed
Weights bf16 (calc.): 14 GB
Where it runs: Not disclosed

Reinforcement learning: Not used · Subgoal image generation + imitation learning
Base control: Fixed · No base control (WidowX·Franka arms)

Key features: Visual chain-of-thought (visual CoT) VLA that first generates a subgoal image. Generating 256 image tokens first makes it on average 7× slower at chunk 10.
Notes: Whether weights are released is unverifiable† (project page links only to arXiv).
Verification: Confirmed