VLA GalleryAll modelsGallery

VLA Gallery / All models / Industry models

GR-2

ByteDance Research · 2024-10

VLAIndustry models230MCVAE

Pretrains future-frame generation on 38M web video clips, then jointly fine-tunes video·actions on robot trajectories.

View architecture diagram in the galleryPaper ↗

Paper

GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot ManipulationarXiv 2410.06158

Overview

Category
Industry models
Type
VLA
Released
2024-10
Organization
ByteDance Research
Open weights
Unverified
Parameters
230M (95M trained); scaling variants trainable parameters 30M (S)/95M (B)/312M (L)/719M (XL)

Architecture

Backbone LLM
None (no VLM)
Vision encoder
VQGAN image tokens
Action head
cVAE action generation → WBC trajectory optimization (CVAE)
Chunk
Not disclosed
Control rate
200 Hz (execution)
Architecture (original summary)
GPT-style Transformer (no VLM): frozen text encoder + VQGAN image tokens; actions generated by cVAE (chunk length not disclosed) → joint commands via WBC trajectory optimization

Training

Pretraining accelerators
Not disclosed
Pretraining duration
Not disclosed
Pretraining steps & batch
Not disclosed
GPU/TPU-hours
Not disclosed
Fine-tuning
Not disclosed
Data type
Human video + public robot data + real robot
Data scale
Pretraining 38M video clips, 50B+ tokens; robot 105 tasks about 40,000 traj (average 400 per task); bin picking 55 objects about 94,000 traj
Main datasets
Howto100M, Ego4D, Something-Something V2, EPIC-KITCHENS, Kinetics-700, RT-1, Bridge
Tasks & evaluation
105 tabletop tasks (8 skills) average 97.7%; 73.9% with only 50 traj per task; bin picking 122 objects 79.0% (GR-1 33.3%); CALVIN 1 task 98.6%, 5 tasks 85.9%
Robot
7-DoF Kinova Gen3 + Robotiq 2F-85

Inference

Inference device & speed
Runs on real robot at 200Hz after WBC trajectory optimization; model inference latency·hardware not disclosed
Inference memory
Not disclosed
Weights bf16 (calc.)
0.5 GB
Where it runs
Not disclosed

Reinforcement learning & base

Reinforcement learning
Not used · Video generation pretraining + imitation learning
Base control
Fixed · No base control (Kinova arm)

Notes

Key features
Pretrains future-frame generation on 38M web video clips, then jointly fine-tunes video·actions on robot trajectories. Experimented with scaling training parameters from 30M to 719M.
Notes
Parameters (B) based on the base model total of 230M (for XL only the 719M trainable parameters are disclosed; total not disclosed). Token count stated as both 'over 50 billion' and 'approximately 50 billion'. No public repository found (weight release could not be confirmed in verification). arXiv v1 2024-10-08.
Verification
Confirmed

Other models in this category