VLA Gallery / All models / Mobile manipulation

VBC

VBC (Visual Whole-Body Control)

UC San Diego · 2024-03

VAMobile manipulationNot disclosedRL policy

A hierarchical quadruped+arm whole-body control framework in which a visual high-level policy outputs base velocity and end-effector pose targets that an RL low-level policy using the leg joints tracks.

Paper

Visual Whole-Body Control for Legged Loco-ManipulationarXiv 2403.16967

Overview

Category

Mobile manipulation

Type

VA

Released

2024-03

Organization

UC San Diego

Open weights

Partial

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

Segmented depth 2-layer CNN (student) · PointNet object encoder (teacher)

Action head

9-dim regression (end-effector deltas 6 · forward/yaw velocity · gripper) → RL low-level policy (RL policy)

Chunk

None (1 step)

Control rate

High level 6–8 Hz, low level 50 Hz

Architecture (original summary)

High-level student: segmented depth CNN (2 layers)+2-layer MLP (teacher uses PointNet object encoder, DAgger distillation) → 9-dim (6 end-effector pose increments·2 forward·yaw velocities·1 gripper), 6–8 Hz / low level: 3-layer MLP (128)+history adaptation module RL policy → 12 leg joint targets 50 Hz, arm via end-effector command IK

Training

Pretraining accelerators

N/A

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Per-task sim training (IsaacGym): high-level teacher RL 10,240 envs about 36 hours, student DAgger 240 envs about 48 hours (single GPU, GTX4090; GTX3090 also used); low-level policy training resources·time not stated

Data type

Simulation·synthetic

Data scale

Sim RL rollouts only (teacher 10,240·student 240 parallel envs), 33 YCB objects; no real data

Main datasets

YCB object models (sim)

Tasks & evaluation

Sim grasping of 33 YCB objects (7 categories): student success rate 0.42–0.74 (teacher 0.76–0.93, non-hierarchical baseline 0.0); real robot 14 objects × 3 heights (ground·box about 0.3 m·table about 0.5 m), 5 trials each, beats baselines in all settings (numbers in Fig. 7 plot)

Robot

Unitree B1 quadruped + Unitree Z1 arm, 2 RealSense D435 (near head·gripper)

Inference

Inference device & speed

High level: 6–8 Hz on external Jetson Orin 64GB (including mask tracking); low level·arm IK: B1 onboard computer, 50 Hz

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Onboard Jetson Orin (high level) + B1 built-in computer (low level)

Reinforcement learning & base

Reinforcement learning

Used (RL controller training) · Both the low-level leg policy and the high-level state-based teacher use sim RL (PPO); the depth student is distilled from the teacher via DAgger

RL evidence

https://arxiv.org/abs/2403.16967

Base control

Quadruped (RL policy drives joints directly) · High-level policy outputs base forward·yaw velocity + end-effector pose increments → RL low-level policy controls leg joints (arm via IK)

Notes

Key features

A hierarchical quadruped+arm whole-body control framework in which a visual high-level policy outputs base velocity and end-effector pose targets that an RL low-level policy using the leg joints tracks. A privileged-state RL teacher is DAgger-distilled into a depth-based student, grasping objects of various heights·shapes on the real robot without real data collection·fine-tuning.

Notes

In real-robot evaluation a human specifies the target object mask with TrackingSAM; real-robot success rates shown only in plots; '81.87' in the teacher table is as written in the source; only code and low-level weights released (GitHub), high-level weights not released; CoRL 2024 Oral.

Verification

Confirmed