VLA Gallery / All models / Mobile manipulation

Learning Visual Quadrupedal Loco-Manipulation from Demonstrations

Shanghai Qi Zhi Institute·HKUST(GZ)·UC Berkeley·Tsinghua University(IIIS) · 2024-03

VAMobile manipulationNot discloseddiffusion

A hierarchical leg-manipulation policy: an RL low-level controller tracks a parameterized Bézier trajectory with one front foot while walking on the other three legs, and a point-cloud-based DP3 diffusion planner outputs the trajectory parameters.

Paper

Learning Visual Quadrupedal Loco-Manipulation from DemonstrationsarXiv 2403.20328

Overview

Category

Mobile manipulation

Type

VA

Released

2024-03

Organization

Shanghai Qi Zhi Institute·HKUST(GZ)·UC Berkeley·Tsinghua University(IIIS)

Open weights

No

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

DP3 point cloud encoder (768 points, world coordinates)

Action head

DP3 diffusion → Bézier trajectory parameters (diffusion)

Chunk

Not disclosed

Control rate

Planner 10 Hz, low-level 50 Hz

Architecture (original summary)

High level: DP3 (768-point world-frame point cloud+proprioception) diffusion → 6th-order rational Bézier control points·weights, start/end pose (SLERP), manipulating-leg flag, 10 Hz / low level: PPO policy (15-step proprioception history) → 12 leg joint targets (low-pass+PD) 50 Hz

Training

Pretraining accelerators

N/A

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Per-task sim training (IsaacGym): low-level PPO 3 billion timesteps, high-level DP3 visual data 20k timesteps; GPU·time not disclosed

Data type

Simulation·synthetic

Data scale

100 sim expert trajectories per task (200 timesteps each, about 3 min to collect); low-level RL 3 billion timesteps

Main datasets

In-house sim expert trajectories (IsaacGym)

Tasks & evaluation

Sim 9 tasks (average of 3 seeds): button 92.3%, door push 85.3%, handle pull 82.3%, basket lift 59.3%, valve turn 52.3%, dishwasher close 50.3%, soccer ball kick 26.0%, object pull 12.7%, dishwasher open 5.7%; real robot qualitative demos only

Robot

Unitree Aliengo quadruped (no arm, manipulates with front legs), RealSense D435 depth camera + external AprilTag localization camera

Inference

Inference device & speed

Low-level 50 Hz, planner 10 Hz; inference hardware not stated (diffusion inference speed noted as a limitation)

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Partial (RL low-level controller) · High-level DP3 planner by BC; low-level foot trajectory tracking·tripod gait controller by IsaacGym PPO (demonstrations also generated in sim with this controller)

RL evidence

https://arxiv.org/abs/2403.20328

Base control

Quadruped (Commands/latents → RL locomotion controller) · DP3 planner outputs foot trajectory parameters in world coordinates → RL controller tracks them while walking on three legs (no separate base velocity command)

Notes

Key features

A hierarchical leg-manipulation policy: an RL low-level controller tracks a parameterized Bézier trajectory with one front foot while walking on the other three legs, and a point-cloud-based DP3 diffusion planner outputs the trajectory parameters. Without an arm, using legs only, it pressed buttons·pushed doors·lifted baskets while moving, in sim and on a real robot (Aliengo).

Notes

Demonstrations are generated not by human teleoperation but by an RL controller tracking designed expert trajectory parameters in sim; the planner does not output base velocity directly but induces motion via world-frame targets (trained within a 4 m×4 m range); no quantitative real-robot results; code·weights not released. Based on: 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations (arXiv 2403.03954) — uses DP3 as the high-level planner backbone (stated in paper)

Verification

Confirmed