VLA Gallery / All models / Autoregressive & latent-action VLA

LAPA

KAIST·University of Washington·Microsoft Research·NVIDIA·Allen Institute for AI · 2024-10

VLAAutoregressive & latent-action VLA7Blatent actions

Pretrains a VLA by learning latent actions from robot·human videos without action labels.

Paper

Latent Action Pretraining from VideosarXiv 2410.11758

Overview

Category

Autoregressive & latent-action VLA

Type

VLA

Released

2024-10

Organization

KAIST·University of Washington·Microsoft Research·NVIDIA·Allen Institute for AI

Open weights

Yes

Parameters

7B (LWM-Chat-1M); LAQ size not disclosed

Architecture

Backbone LLM

LWM-Chat-1M 7B (Llama 2 7B-based)

Vision encoder

LWM vision tokenizer

Action head

Latent action tokens → action decoder (latent actions)

Chunk

Not disclosed

Control rate

Not disclosed

Architecture (original summary)

LWM-Chat-1M 7B; latent action tokens (LAQ VQ-VAE, vocabulary 8·length 4, frame 0.6 s ahead) → real actions in fine-tuning

Training

Pretraining accelerators

8 H100

Pretraining duration

34 h

Pretraining steps & batch

batch 128 (paper)/256·70K steps (README)

GPU/TPU-hours

272 (Stated · H100-hours)

Fine-tuning

4 × A100 80GB (README), until training action accuracy reaches 95%; time not disclosed

Data type

Real-robot teleoperation (video without actions) + human video

Data scale

OXE 970k demos (actions not used) + SSv2 220K videos; Bridgev2 60k·LangTable 181k trajectories

Main datasets

Open X-Embodiment, Something-Something V2, BridgeData V2, Language Table

Tasks & evaluation

LangTable; SIMPLER (WidowX, multitask fine-tuning on 100 trajectories); 3 real tabletop tasks; real average 50.1% (OpenVLA 43.9%)

Robot

LangTable simulation, WidowX (SIMPLER), real tabletop robot

Inference

Inference device & speed

Not disclosed

Inference memory

Not disclosed

Weights bf16 (calc.)

14 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Not used · Latent action pretraining + imitation learning fine-tuning

Base control

Fixed · No base control (tabletop arm)

Notes

Key features

Pretrains a VLA by learning latent actions from robot·human videos without action labels. Claims 30–40× efficiency over OpenVLA at 272 H100-hours.

Notes

The 30~40× efficiency rests on the authors' assumption that H100 is 2~3× faster than A100. LAQ training compute is not disclosed.

Verification

Confirmed