VLA Gallery / All models / Mobile manipulation

QUART-Online

Westlake University(MiLAB)·Zhejiang University·Beijing University of Posts and Telecommunications · 2024-12

VLAMobile manipulation8Bdiscrete tokens

A quadruped VLA that removes latency without shrinking parameters by fine-tuning an MLLM, with ACD compressing continuous action chunks into residual VQ codes, to output multi-step commands in one inference.

Paper

QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot LearningarXiv 2412.15576

Overview

Category

Mobile manipulation

Type

VLA

Released

2024-12

Organization

Westlake University(MiLAB)·Zhejiang University·Beijing University of Posts and Telecommunications

Open weights

Yes

Parameters

8B (paper Table I)/9B (HF metadata)

Architecture

Backbone LLM

Persimmon decoder · Fuyu-8B config (README)

Vision encoder

Fuyu-8B patch tokenization (no separate vision encoder, README)

Action head

ACD residual VQ action tokens → decoded into 10-step command chunks (discrete tokens)

Chunk

10 steps (-1 · -5 variants also tested)

Control rate

Effective 50 Hz

Architecture (original summary)

Fuyu-8B MLLM (image + language) → ACD compressed action tokens (residual VQ, codebook 512 · 2 quantizers, 1D conv encoder) → decoded into a 10-step continuous command chunk (termination · vx · vy · yaw · body height · frequency · gait 3 · foot height · pitch · stance width), effective 50 Hz → Walk These Ways low-level controller

Training

Pretraining accelerators

4 GPUs (README training script; model not stated)

Pretraining duration

Not disclosed

Pretraining steps & batch

100K steps · batch 256 · 10 epochs (paper)/batch 128 = 32 × 4 GPUs (README)

GPU/TPU-hours

Not disclosed

Fine-tuning

N/A; ACD RVQ tokenizer trained separately (lr 3e-4; README: batch 1024, 50 epochs), resources and time not disclosed

Data type

Simulation·synthetic

Data scale

Uses QUARD (size not stated in paper); 50 Hz data downsampled to 5 Hz (README)

Main datasets

QUARD

Tasks & evaluation

QUARD sim (50 trials per task) average success unseen visual 0.68·unseen language 0.79 vs QUART 0.37·0.52; 65% success improvement per abstract; real-robot experiments shown in figures only, no numbers reported

Robot

IsaacGym quadruped sim (QUARD); real robot model not stated

Inference

Inference device & speed

Control frequency = chunk length × MLLM inference frequency (Eq. 10), QUART-Online-10 50 Hz (Table II); README: A100 fp16 about 20 ms per action, VRAM fp16 >19GB, fp32 >37GB

Inference memory

>19GB (fp16) / >37GB (fp32), README

Weights bf16 (calc.)

16 GB

Where it runs

GPU server (A100 per README)

Reinforcement learning & base

Reinforcement learning

Partial (RL low-level controller) · MLLM·ACD use supervised learning only; commands executed by the low-level Walk These Ways controller (PPO in the original paper; RL not stated in this paper)

RL evidence

https://arxiv.org/abs/2212.03238

Base control

Quadruped (Commands/latents → RL locomotion controller) · VLA outputs 10-step high-level command chunks (RVQ tokens) → Walk These Ways controller controls joints at the low-level control rate (50 Hz)

Notes

Key features

A quadruped VLA that removes latency without shrinking parameters by fine-tuning an MLLM, with ACD compressing continuous action chunks into residual VQ codes, to output multi-step commands in one inference. Matching QUART (previously 2 Hz) to the 50 Hz low-level control rate, it raised task success rate by 65%.

Notes

Shows sharp success drops when shrinking parameters (5.3B, 2.7B) (0.74→0.22, 0.11) and proposes ACD as an alternative; no real-robot details; batch (256 vs 128) and parameters (8B vs 9B) differ by source; code (MIT), weights, and QUARD data released on HF; ICRA 2025 (project page). Based on: QUAR-VLA: Vision-Language-Action Model for Quadruped Robots (arXiv 2312.14457) — low-latency successor to the 2 Hz QUART (stated in the paper), uses QUARD data

Verification

Confirmed