VLA Gallery / All models / VA

DP3

Shanghai Qi Zhi Institute; Shanghai Jiao Tong University; Tsinghua University(IIIS); Shanghai AI Lab · 2024-03

VAVANot discloseddiffusion

3D diffusion policy that encodes sparse point clouds with a small MLP and learns from about 10 demonstrations per task.

Paper

3D Diffusion PolicyarXiv 2403.03954

Overview

Category

VA

Type

VA

Released

2024-03

Organization

Shanghai Qi Zhi Institute; Shanghai Jiao Tong University; Tsinghua University(IIIS); Shanghai AI Lab

Open weights

No

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

Point cloud MLP encoder (64-dim)

Action head

CNN diffusion policy (DDIM 10 steps) (diffusion)

Chunk

4 steps (3 executed)

Control rate

Not disclosed

Architecture (original summary)

Point cloud MLP encoder (3 layers, channels 3, 64, 128, 256, max-pool, 64-dim projection) + CNN-based diffusion policy; horizon H 4, 2 observation and 3 execution steps; DDIM 100 training and 10 inference steps; 512–1,024 points

Training

Pretraining accelerators

N/A

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Per task: paper does not disclose GPU·time (MetaWorld 1000 epochs, others 3000 epochs, batch 128); README: 1× A40 about 3 hours·GPU memory about 10G, Simple DP3 1–2 hours

Data type

Simulation·synthetic; real-robot teleoperation

Data scale

Sim: 10 demos per task for most tasks (DexArt·HORA 100); real: 40 demos per task

Main datasets

Adroit; Bi-DexHands; DexArt; DexDeform; DexMV; HORA; MetaWorld; in-house real data

Tasks & evaluation

Sim 72 tasks across 7 domains (Adroit, Bi-DexHands, DexArt, DexDeform, DexMV, HORA, MetaWorld); real Roll-Up, Dumpling, Drill, Pour

Robot

Franka arm + Allegro hand · gripper, RealSense L515

Inference

Inference device & speed

DP3 12.7 FPS, DP 12.3 FPS, Simple DP3 25.3 FPS (hardware not stated)

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning (behavior cloning) only

Base control

Fixed · No base control (Franka arm + hand·gripper)

Notes

Key features

3D diffusion policy that encodes sparse point clouds with a small MLP and learns from about 10 demonstrations per task. Covers 72 sim tasks across 7 domains and real tasks with a dexterous hand.

Notes

A40 training time is from the official README, not the paper. Code and real data released, no checkpoints.

Verification

Confirmed