VLA Gallery / All models / Mobile manipulation

HarmonicMM

UC San Diego·Allen Institute for AI(PRIOR)·University of Washington · 2023-12

VAMobile manipulationNot disclosedRL policy

Trains a single policy end-to-end with DD-PPO in ProcTHOR sim that moves base and arm together from RGB alone, without separating navigation and manipulation modules.

Paper

Harmonic Mobile ManipulationarXiv 2312.06639

Overview

Category

Mobile manipulation

Type

VA

Released

2023-12

Organization

UC San Diego·Allen Institute for AI(PRIOR)·University of Washington

Open weights

No

Parameters

Not disclosed (DINOv2 ViT-S/14 encoder + GRU 512)

Architecture

Backbone LLM

None

Vision encoder

DINOv2 ViT-S/14 (2 Nav · Manip cameras)

Action head

Actor-critic FC (5-D continuous actions + grasp) (RL policy)

Chunk

None (1 step)

Control rate

Not disclosed

Architecture (original summary)

Pretrained DINOv2 (dinov2_vits14) features (2 RGB cameras, Nav·Manip) → per-camera 2-layer 1×1 Conv (16 channels) + proprioception → 1-layer GRU (512) → policy·value FC (512); 5-D continuous actions (base translation·rotation, arm height·extension, wrist rotation; door·fridge tasks add 1-D grasp), no chunking

Training

Pretraining accelerators

N/A

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Per-task RL training: DD-PPO (AllenAct) 10^7 steps, 20 parallel environments·640 samples per iteration, 3 seeds; GPU·training time not disclosed

Data type

Simulation·synthetic

Data scale

ProcTHOR houses: 700 doors·2,000 tables·1,400 fridges; 10^7 env steps per task

Main datasets

ProcTHOR (AI2-THOR) procedurally generated houses

Tasks & evaluation

Sim (350 unseen houses) success rates: Cleaning Table 36.7%, Opening Door Push 80.0%·Pull 51.2%, Opening Fridge 20.0% (average 47%; ReLMoGen 25.1%·51.7%·40.7%·0%); real world: door pull 9/15, push 7/10, table cleaning 8/12 (surface 66.6%)

Robot

Hello Robot Stretch (wheeled base, arm height·extension control), RealSense D455 RGB

Inference

Inference device & speed

Not disclosed

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Used (RL controller training) · Entire policy trained from scratch per task with DD-PPO in ProcTHOR sim (10^7 steps), then transferred zero-shot to the real world

RL evidence

https://arxiv.org/abs/2312.06639

Base control

Wheeled (RL policy drives joints directly) · Policy directly outputs base translation and rotation together with arm height, extension, and wrist rotation as 5-dim continuous actions

Notes

Key features

Trains a single policy end-to-end with DD-PPO in ProcTHOR sim that moves base and arm together from RGB alone, without separating navigation and manipulation modules. Averaged 47% success on 4 sim tasks, beating baselines such as ReLMoGen, and performed door opening·table cleaning in unseen real apartments without fine-tuning.

Notes

Pre-2024 reference baseline (anchor). A separate policy per task with no language or goal input. The vision encoder is a ViT (DINOv2) but the policy trunk is a GRU, so classified as non-Transformer. Real-robot door handle grasping is heuristic-assisted. Code released (GitHub), no checkpoints. IROS 2024 Best Paper on Mobile Manipulation.

Verification

Confirmed