VLA Gallery / All models / Humanoid whole-body controllers

OmniH2O

Carnegie Mellon University·Shanghai Jiao Tong University · 2024-06

VAHumanoid whole-body controllersNot disclosedRL policy

A tracker for whole-body teleoperation via VR·RGB keypoints, and a platform for training autonomous policies from the collected demonstrations.

Paper

OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and LearningarXiv 2406.08858

Overview

Category

Humanoid whole-body controllers

Type

VA

Released

2024-06

Organization

Carnegie Mellon University·Shanghai Jiao Tong University

Open weights

Unverified

Parameters

Not disclosed (MLP [512, 256, 128])

Architecture

Backbone LLM

None

Vision encoder

—

Action head

Joint targets (RL policy)

Chunk

Not disclosed

Control rate

50 Hz (PD 200 Hz)

Architecture (original summary)

MLP [512, 256, 128] (GRU·LSTM student variants tested), student 25-step history; VR or RGB keypoint input → joint targets, policy 50 Hz·PD 200 Hz

Training

Pretraining accelerators

Not disclosed (Isaac Gym preview 4, repo)

Pretraining duration

Not disclosed

Pretraining steps & batch

4096 env (repo training command)

GPU/TPU-hours

Not disclosed

Fine-tuning

N/A

Data type

Motion capture / real-robot teleoperation

Data scale

Retargeted·augmented AMASS 14k sequences (sim evaluation set); OmniH2O-6 real teleoperation demonstrations totaling 40 min (8,535~18,436 frames per task)

Main datasets

AMASS (retargeting · augmentation), OmniH2O-6

Tasks & evaluation

Sim tracking metrics; real autonomous 6 tasks (OmniH2O-6, Diffusion Policy·ResNet BC)

Robot

Unitree H1 + hands

Inference

Inference device & speed

Policy 50 Hz, PD controller 200 Hz

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Used (RL controller training) · RL teacher (privileged information) → DAgger student distillation

Base control

Humanoid legged (RL policy drives joints directly) · Whole-body RL policy takes keypoint/joystick goals and outputs joint targets including legs (joystick forward/backward walking, 'Walk forward 0.5m')

Notes

Key features

A tracker for whole-body teleoperation via VR·RGB keypoints, and a platform for training autonomous policies from the collected demonstrations. Also demonstrated autonomous behavior linked with GPT-4o.

Notes

README commands also include short_history_length 0/5/50 variants. Code CC BY-NC 4.0, checkpoints unverified. arXiv v1 2024-06-13, CoRL 2024. Data hours null because given in minutes (40 minutes). Verification covered only repo values (Isaac Gym·env·license·history).

Verification

Confirmed