VLA GalleryAll modelsGallery

VLA Gallery / All models / VA

BAKU

New York University · 2024-06

VAVAAbout 10Mregression

Trains an about 10M-parameter multitask policy on one consumer GPU in 6–26 h per benchmark.

View architecture diagram in the galleryPaper ↗

Paper

BAKU: An Efficient Transformer for Multi-Task Policy LearningarXiv 2406.07539

Overview

Category
VA
Type
VA
Released
2024-06
Organization
New York University
Open weights
No
Parameters
About 10M (encoder 2.1M + trunk 6.5M + head 1.4M); Table 3 variants 4.4M·10M·31M·114M

Architecture

Backbone LLM
None
Vision encoder
FiLM ResNet-18
Action head
MLP head + chunk smoothing (regression)
Chunk
10 (LIBERO) · 3 (DMC)
Control rate
10 Hz
Architecture (original summary)
FiLM-conditioned ResNet-18 vision encoder + proprioception → Transformer observation trunk; MLP action head + action chunks (LIBERO · Meta-World 10, DMC 3) · temporal smoothing, uses only the latest observation, 10 Hz

Training

Pretraining accelerators
N/A
Pretraining duration
N/A
Pretraining steps & batch
N/A
GPU/TPU-hours
Not disclosed
Fine-tuning
Per-benchmark multi-task models, 1× RTX A4000: LIBERO-90 600k steps·batch 64 about 10.5 hours, Meta-World 600k·64 about 8 hours, DMC 2M·128 about 26 hours, xArm 200k·64 about 6 hours
Data type
Simulation·synthetic; real-robot teleoperation
Data scale
50 (LIBERO-90)·35 (Meta-World)·500 (DMC) demos per task; real total 520 demos (average 17 per task)
Main datasets
LIBERO-90; Meta-World; DeepMind Control; in-house xArm
Tasks & evaluation
LIBERO-90 (90 tasks), Meta-World 30 tasks, DeepMind Control 9 locomotion tasks, real xArm 30 tasks
Robot
xArm single arm; simulation

Inference

Inference device & speed
Real xArm 10 Hz closed loop; latency not disclosed
Inference memory
Not disclosed
Weights bf16 (calc.)
0.2 GB
Where it runs
Not disclosed

Reinforcement learning & base

Reinforcement learning
Not used · Imitation learning (behavior cloning) only
Base control
Fixed · No base control (xArm single arm·sim)

Notes

Key features
Trains an about 10M-parameter multitask policy on one consumer GPU in 6–26 h per benchmark. Tasks are specified by frozen MiniLM sentence embeddings or goal images.
Notes
A borderline language-conditioned case using frozen MiniLM text task embeddings (no VLM backbone). Parameters (B) are the largest variant, 114M. Training time refers to per-benchmark multi-task models, not per task. Includes DMC locomotion tasks.
Verification
Confirmed

Other models in this category