VLA Gallery / All models / Autoregressive & latent-action VLA

RoboFlamingo

ByteDance Research·Tsinghua University·SJTU·NUS · 2023-11

VLAAutoregressive & latent-action VLAOpenFlamingo 3B/4B/9Bregression

A continuous-action VLA that freezes the VLM and trains only the resampler·cross-attention·policy head.

Paper

Vision-Language Foundation Models as Effective Robot ImitatorsarXiv 2311.01378

Overview

Category

Autoregressive & latent-action VLA

Type

VLA

Released

2023-11

Organization

ByteDance Research·Tsinghua University·SJTU·NUS

Open weights

Yes

Parameters

OpenFlamingo 3B/4B/9B; 1B trainable parameters (Table 2)

Architecture

Backbone LLM

MPT·RedPajama family · OpenFlamingo 3B/9B

Vision encoder

CLIP ViT-L/14

Action head

LSTM policy head (continuous 6-DoF) (regression)

Chunk

Not disclosed

Control rate

Not disclosed

Architecture (original summary)

OpenFlamingo (CLIP ViT-L/14 + MPT·RedPajama LLM) + LSTM policy head; continuous actions (6-DoF relative pose MSE + gripper BCE)

Training

Pretraining accelerators

N/A (uses OpenFlamingo VLM weights)

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

1 server with 8 × A100 (80G), batch 6 per GPU; MPT-3B 13 h per epoch (best at epoch 3), MPT-9B 26 h per epoch (best at epoch 4)

Data type

Simulation·synthetic

Data scale

CALVIN play data, 1% language annotation (about 24k steps)

Main datasets

CALVIN

Tasks & evaluation

CALVIN ABCD→D average length 4.09 (HULC 3.06, RT-1 2.45); ABC→D 2.475 (README)

Robot

CALVIN simulation

Inference

Inference device & speed

Hz not disclosed; open-loop action sequence prediction reduces compute

Inference memory

Not disclosed

Weights bf16 (calc.)

18 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning (behavior cloning) only

Base control

Fixed · No base control (CALVIN arm sim)

Notes

Key features

A continuous-action VLA that freezes the VLM and trains only the resampler·cross-attention·policy head. Trained without robot pretraining on CALVIN data with only 1% language annotation.

Notes

Announced 2023-11, outside the 2024~2025 range, but included as an assigned model. A continuous regression VLA rather than discrete tokens; parameters (B) are based on the largest 9B variant.

Verification

Confirmed