VLA Gallery / All models / Mobile manipulation

RoboDuet

Tsinghua University·Shanghai Qi Zhi Institute·Shanghai AI Laboratory·UC San Diego·Jianghuai Advance Technology Center · 2024-03

기타Mobile manipulationNot disclosedRL policy

A two-stage-trained whole-body control framework in which two RL policies, a locomotion policy and an arm policy, cooperate (the arm policy commands body pitch·roll) to perform quadruped+arm base movement and end-effector 6D pose tracking simultaneously.

Paper

RoboDuet: Learning a Cooperative Policy for Whole-body Legged Loco-ManipulationarXiv 2403.17367

Overview

Category

Mobile manipulation

Type

기타

Released

2024-03

Organization

Tsinghua University·Shanghai Qi Zhi Institute·Shanghai AI Laboratory·UC San Diego·Jianghuai Advance Technology Center

Open weights

No

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

—

Action head

12 leg·6 arm joint targets + body pitch·roll commands (RL policy)

Chunk

None (1 step)

Control rate

50 Hz

Architecture (original summary)

Two MLP (512·256·128, ELU) PPO policies for locomotion·arm; locomotion policy → 12 leg joint targets, arm policy → 6 arm joint offsets + 2 body pitch·roll commands, 50 Hz; input is proprioception + velocity command + 9-dim end-effector target

Training

Pretraining accelerators

N/A

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Sim RL (IsaacGym, PPO asymmetric actor-critic): 4,096 parallel agents, stage 1 10k + stage 2 40k iter (3 seeds), RTX 4090; training time not disclosed

Data type

Simulation·synthetic

Data scale

Sim RL rollouts with 4,096 parallel agents, 50k iter total

Main datasets

None (sim RL)

Tasks & evaluation

Real-robot object carrying (heights 0·20·60·100 cm) success 39/60 (AFB+ID 32/60, FB+IK 10/60), 23% higher success rate (as stated in paper); extreme 6D pose self-collision rate 5% (FB+IK 50%); cross-transfer in sim Go1·Go2·A1+ARX5

Robot

Unitree Go1 Edu + ARX5 6-DoF arm (parallel gripper, RealSense D435i above gripper); zero-shot transfer to Go2+ARX5

Inference

Inference device & speed

Policy 50 Hz; Go2 deployment code runs on Jetson Orin or a PC (README), Go1 deployed via Docker

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Onboard (Go2: Jetson Orin or PC, README)

Reinforcement learning & base

Reinforcement learning

Used (RL controller training) · The entire cooperative locomotion·arm policy trained with IsaacGym PPO (asymmetric actor-critic, two stages); no high-level policy (teleoperation commands)

RL evidence

https://arxiv.org/abs/2403.17367

Base control

Quadruped (RL policy drives joints directly) · Teleoperated base velocity and end-effector pose commands → cooperative RL policy (locomotion, arms) directly outputs leg and arm joint targets

Notes

Key features

A two-stage-trained whole-body control framework in which two RL policies, a locomotion policy and an arm policy, cooperate (the arm policy commands body pitch·roll) to perform quadruped+arm base movement and end-effector 6D pose tracking simultaneously. Showed 23% higher task success than baselines and zero-shot cross-embodiment transfer from Go1 to Go2.

Notes

A proprioceptive RL controller with no visual input; in experiments base velocity·end-effector pose are commanded by VR teleoperation (Meta Quest 3) (no high-level learned policy); the calculation basis of 23% vs 39/60 and 32/60 is unclear; the paper lists the affiliation as 'University of San Diego' (project page: UC San Diego); code (based on Walk These Ways)·deployment code released, weights not released; RA-L 2025. Based on: Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior (arXiv 2212.03238) — README states the base implementation was borrowed from walk-these-ways

Verification

Confirmed