VLA Gallery / All models / Humanoid whole-body controllers

iDP3

iDP3 (Improved 3D Diffusion Policy)

Stanford·SFU·UPenn·UIUC·CMU · 2024-10

VAHumanoid whole-body controllersNot discloseddiffusion

An egocentric 3D point cloud diffusion policy that performs humanoid upper-body manipulation at 15 Hz using only onboard compute.

Paper

Generalizable Humanoid Manipulation with Improved 3D Diffusion PoliciesarXiv 2410.10803

Overview

Category

Humanoid whole-body controllers

Type

VA

Released

2024-10

Organization

Stanford·SFU·UPenn·UIUC·CMU

Open weights

Yes

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

Pyramid convolutional point cloud encoder

Action head

Diffusion policy (DDIM 10 steps) (diffusion)

Chunk

16 steps

Control rate

15 Hz

Architecture (original summary)

Pyramid convolutional encoder + diffusion policy (DDIM 50 training steps, 10 inference steps), prediction horizon 16; egocentric 3D point cloud (RealSense L515, 4096 points at inference) + proprioception → target joint positions, 15 Hz

Training

Pretraining accelerators

RTX 4090 × 1 (README)

Pretraining duration

Not disclosed

Pretraining steps & batch

300 epochs (AdamW)

GPU/TPU-hours

Not disclosed

Fine-tuning

N/A

Data type

Real-robot teleoperation

Data scale

10 demonstrations × 10 rollouts per task, 300 episodes total over 3 tasks (20 successes per demonstration); single training scene

Main datasets

Apple Vision Pro upper-body teleoperation data (in-house)

Tasks & evaluation

Training scene Pick&Place 9/10, Pour 9/10, Wipe 10/10; new objects, viewpoints, scenes 9/10; over 2000 real-world evaluations (Table I 2253 episodes)

Robot

Fourier GR1 (25-DoF upper body on a height-adjustable cart)

Inference

Inference device & speed

Real-time 15 Hz with onboard robot compute only

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Onboard

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning (behavior cloning) only

Base control

Fixed · Lower body disabled, locomotion replaced by a cart (policy outputs only upper-body 25-DoF joint targets)

Notes

Key features

An egocentric 3D point cloud diffusion policy that performs humanoid upper-body manipulation at 15 Hz using only onboard compute. Uses a pyramid convolutional encoder, based on evidence from human data that convolutional layers produce smoother actions.

Notes

No sentence describes the diffusion backbone as a transformer, so classified as non-transformer based on the convolutional encoder. Data and 3-task checkpoints released 2024-11-04 (README). Verification covered only README values (training GPU, release date).

Verification

Confirmed