VLA Gallery / All models / Autoregressive & latent-action VLA

SpatialVLA

Shanghai AI Lab·Fudan University·SJTU·Zhejiang University·ShanghaiTech·Northwestern Polytechnical University · 2025-01

VLAAutoregressive & latent-action VLA3.5Bdiscrete tokens

Reduces 7-dim actions to 3 tokens with 3D positional encoding and adaptive grid tokens.

Paper

SpatialVLA: Exploring Spatial Representations for Visual-Language-Action ModelarXiv 2501.15830

Overview

Category

Autoregressive & latent-action VLA

Type

VLA

Released

2025-01

Organization

Shanghai AI Lab·Fudan University·SJTU·Zhejiang University·ShanghaiTech·Northwestern Polytechnical University

Open weights

Yes

Parameters

3.5B (paper)/4B (checkpoint name)

Architecture

Backbone LLM

Gemma 2 2B · PaliGemma2 3B config

Vision encoder

SigLIP + Ego3D depth encoding

Action head

Adaptive grid spatial tokens (3 per action) (discrete tokens)

Chunk

4 steps

Control rate

Not disclosed

Architecture (original summary)

PaliGemma2 (paligemma2-3b-pt-224) + Ego3D position encoding (ZoeDepth); adaptive action grids give 3 spatial tokens per action (vocabulary 8194), chunk T=4 (12 tokens)

Training

Pretraining accelerators

64 × A100

Pretraining duration

10 days (paper)/about 10 days (README)

Pretraining steps & batch

stage 1 160k + stage 2 40k steps (after excluding DROID), batch 2048, lr 2e-5

GPU/TPU-hours

15,360 (Converted · A100-hours (64 × 240 h))

Fine-tuning

LoRA, 4 or 8 A100 (README); LIBERO uses LoRA r=32·200 epochs; time not disclosed

Data type

Real-robot teleoperation

Data scale

1.1M episodes (28 datasets); 50–200 demos per Franka fine-tuning task

Main datasets

OXE (Bridge 15.34%, Fractal 14.71%, DROID 11.66%, BC-Z 8.64%, etc.), RH20T 5.67%

Tasks & evaluation

SimplerEnv Google Robot zero-shot visual matching 71.9% (RT-2-X 60.7%); WidowX zero-shot 34.4%·fine-tuned 42.7%; LIBERO average 78.1%; real Franka single-task 82%·multi-task 57%

Robot

WidowX, Google robot (SimplerEnv), Franka

Inference

Inference device & speed

Single RTX 4090 about 20 Hz, 8.5 GB

Inference memory

8.5 GB (RTX 4090)

Weights bf16 (calc.)

7 GB

Where it runs

Local PC

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning (behavior cloning) only

Base control

Fixed · No base control (arm only: EE actions)

Notes

Key features

Reduces 7-dim actions to 3 tokens with 3D positional encoding and adaptive grid tokens. Runs at about 20 Hz on RTX 4090.

Notes

Verification corrected affiliations (no TeleAI, added Northwestern Polytechnical University). Parameters (B) is the paper value of 3.5B.

Verification

Corrections applied