VLA Gallery / All models / Mobile manipulation

Helpful DoggyBot

Stanford University·UC San Diego · 2024-09

기타Mobile manipulationNot disclosedRL policy

Modular indoor object-fetching system for a Go2 with a front-bottom 1-DoF gripper, combining an egocentric depth-based agile controller trained with sim RL and pretrained VLM perception with proportional control.

Paper

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language ModelsarXiv 2410.00231

Overview

Category

Mobile manipulation

Type

기타

Released

2024-09

Organization

Stanford University·UC San Diego

Open weights

No

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

Depth CNN+GRU (controller) · Florence-2·SAM2 · GroundingDINO·MobileSAM·Cutie (perception)

Action head

RL policy joint targets → Unitree PD (RL policy)

Chunk

None (1 step)

Control rate

50 Hz (depth 10 Hz)

Architecture (original summary)

Learned controller: depth CNN+GRU estimator (ROA distillation) + RL policy → joint targets (sent with PD gains) at 50 Hz, depth encoder at 10 Hz; commands = linear velocity, angular velocity, pitch; high level is Florence-2, SAM2 (ceiling camera), GroundingDINO, MobileSAM, Cutie perception + proportional control

Training

Pretraining accelerators

N/A

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Sim RL (IsaacGym Preview 4): privileged (oracle) policy 6,144 robots·400 terrains 20k iter 10 h (1 RTX 4090), depth distillation 384 robots 5k iter 6 h; VLM used as the pretrained model unchanged

Data type

Simulation·synthetic

Data scale

Sim RL rollouts (6,144·384 parallel robots); no real data

Main datasets

None (400 sim terrains)

Tasks & evaluation

First-attempt success rate in 2 unseen real environments (10 trials each): bed+toy 60%, sofa+bottle 60%, floor+ball 70%; sim metrics averaged over 200 trials

Robot

Unitree Go2 quadruped (12-DoF) + lower-front 1-DoF gripper (Dynamixel), egocentric RealSense D435, ceiling fisheye camera

Inference

Inference device & speed

Controller: onboard Jetson (Orin) 50 Hz, depth encoder 10 Hz; VLM: separate workstation (SAM2 tracking 10 Hz); grasp perception (GroundingDINO, MobileSAM 0.2 Hz, Cutie 10 Hz) runs onboard (on-device)

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Onboard Jetson (controller) + workstation (VLM)

Reinforcement learning & base

Reinforcement learning

Used (RL controller training) · The only learned policy, a low-level depth controller, is trained via PPO (privileged teacher)→ROA depth student; high-level perception·commands are not learned

RL evidence

https://arxiv.org/abs/2410.00231

Base control

Quadruped (Commands/latents → RL locomotion controller) · From VLM-detected position, a proportional controller generates linear velocity (0.8 m/s)·yaw·pitch commands → depth-based RL policy controls joints

Notes

Key features

Modular indoor object-fetching system for a Go2 with a front-bottom 1-DoF gripper, combining an egocentric depth-based agile controller trained with sim RL and pretrained VLM perception with proportional control. Without collecting or training on real data, it achieved 60–70% success in two unseen homes, including climbing onto a bed to fetch a toy.

Notes

Only the low-level RL controller is learned; detection and tracking use pretrained models, and motion and grasp commands use rule-based proportional control (linear velocity fixed at 0.8 m/s); reliance on a ceiling camera stated as a limitation; natural-language object specification via Florence-2 prompts; code released (GitHub), weights not released.

Verification

Corrections applied