VLA Gallery / All models / Mobile manipulation

QUAR-VLA (QUART)

Westlake University·Zhejiang University · 2023-12

VLAMobile manipulation8Bdiscrete tokens

Proposed the quadruped VLA task definition, in which quadruped high-level commands (velocity·gait·pose) are output as discrete tokens from a single RGB image and language instruction, along with the QUART model and the sim·real-robot QUARD dataset.

Paper

QUAR-VLA: Vision-Language-Action Model for Quadruped RobotsarXiv 2312.14457

Overview

Category

Mobile manipulation

Type

VLA

Released

2023-12

Organization

Westlake University·Zhejiang University

Open weights

No

Parameters

8B (based on Fuyu-8B; v1's QUART-1 is about 30M)

Architecture

Backbone LLM

Fuyu-8B (decoder-only VLM; no separate LLM named)

Vision encoder

Fuyu-8B takes image patches directly (no separate vision encoder)

Action head

256-bin discrete tokens (11-dim command + termination) (discrete tokens)

Chunk

None (1 step)

Control rate

2 Hz

Architecture (original summary)

Fuyu-8B VLM (single RGB + language) → 256-bin discrete tokens: 11-dim high-level command (vx · vy · ωz, gait phase 3, frequency, body height, pitch, stance width, foot height) + termination, no chunking, 2 Hz → Walk These Ways command-tracking controller generates joints

Training

Pretraining accelerators

Not disclosed

Pretraining duration

Not disclosed

Pretraining steps & batch

100K steps · batch 256 (lr 2e-5)

GPU/TPU-hours

Not disclosed

Fine-tuning

N/A (single stage of joint training on QUARD sim and real data)

Data type

Simulation·synthetic / real-robot teleoperation

Data scale

QUARD total 259K episodes (table total·Fig. 2; summing table rows gives sim 246K+real 3K=249K), real robot 3K (Go to Object, teleoperated); v1 has 348K sim

Main datasets

QUARD

Tasks & evaluation

QUARD sim success rates: Distinguish 0.66, Go to 0.60, Go avoid 0.53, Go through 0.41, Crawl 0.32, Unload 0.12, unseen objects 0.35·unseen language 0.33; real robot Go to 13/20 (sim 256K+real 3K) vs 3/20 (real 3K only); 4,000 evaluations total

Robot

WR-2 quadruped (12 joints, as named in paper) + front RealSense D435; sim IsaacGym

Inference

Inference device & speed

Real world 2 Hz (v6; v1: QUART-1 5 Hz·QUART-2 1 Hz), inference hardware not stated

Inference memory

Not disclosed

Weights bf16 (calc.)

16 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Partial (RL low-level controller) · QUART uses supervised fine-tuning only (100K steps); low-level command tracking by the Walk These Ways policy (PPO in the original paper; RL not stated in this paper)

RL evidence

https://arxiv.org/abs/2212.03238

Base control

Quadruped (Commands/latents → RL locomotion controller) · VLA outputs 11-dim high-level command tokens → Walk These Ways command-following controller controls joints

Notes

Key features

Proposed the quadruped VLA task definition, in which quadruped high-level commands (velocity·gait·pose) are output as discrete tokens from a single RGB image and language instruction, along with the QUART model and the sim·real-robot QUARD dataset. 4,000 evaluations showed that co-training on large-scale sim data and a small amount of real data raised real-robot Go to success from 3/20 to 13/20.

Notes

Sim data generated by an A*/D* path + PD velocity conversion script; real data by remote control; low-level controller [23] = Walk These Ways (this paper does not state RL; the original paper uses PPO); training GPU·time not stated; a LLaVA-7B variant was also compared (similar performance); model configuration·data numbers changed across v1→v6; weights not released (QUARD released on HF via the QUART-Online page).

Verification

Confirmed