VLA Gallery / All models / Autoregressive & latent-action VLA

LLaRA

Stony Brook University·University of Wisconsin–Madison · 2024-06

VLAAutoregressive & latent-action VLA7Bdiscrete tokens

Converts robot actions to text in LLaVA conversation format and trains with 6 kinds of auxiliary self-supervised data.

Paper

LLaRA: Supercharging Robot Learning Data for Vision-Language PolicyarXiv 2406.20095

Overview

Category

Autoregressive & latent-action VLA

Type

VLA

Released

2024-06

Organization

Stony Brook University·University of Wisconsin–Madison

Open weights

Yes

Parameters

7B(LLaVA-1.5-7B)

Architecture

Backbone LLM

Vicuna 7B · LLaVA-1.5 config

Vision encoder

CLIP ViT-L (frozen)

Action head

Text coordinate and rotation output (discrete tokens)

Chunk

Not disclosed

Control rate

Not disclosed

Architecture (original summary)

LLaVA-1.5-7B (vision encoder frozen); actions output as text (normalized 2D image coordinates + rotation angle)

Training

Pretraining accelerators

N/A (no robot pretraining)

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Paper: batch 32 (VIMA-0.8k/8k)/128 (VIMA-80k), lr up to 2e-5, GPU·time Not disclosed; README: full FT needs 4 Ampere GPUs with 24GB+ + 300GB RAM (verified on 4× A5000), LoRA needs 1× 24GB GPU

Data type

Simulation·synthetic + real robot (xArm)

Data scale

VIMA expert trajectories 0.8k/8k/80k; auxiliary·xArm sample counts not disclosed

Main datasets

VIMA-Bench, xArm-Det, xArm-Action

Tasks & evaluation

VIMA-Bench L1–L3 (20 seeds); real xArm (20 episodes per setting)

Robot

VIMA-Bench simulation, xArm

Inference

Inference device & speed

Single 24GB GPU (README); speed not disclosed

Inference memory

Runs on a single 24 GB GPU (README)

Weights bf16 (calc.)

14 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Not used · Instruction tuning (supervised learning)

Base control

Fixed · No base control (xArm arm·sim)

Notes

Key features

Converts robot actions to text in LLaVA conversation format and trains with 6 kinds of auxiliary self-supervised data. Fine-tunes the VLM directly without robot pretraining.

Notes

Hardware is a README requirement, not from the paper.

Verification

Confirmed