VLA Gallery / All models / Reference baselines (pre-2024)

Diffusion Policy (Transformer)

Columbia University; Toyota Research Institute; MIT; Stanford University · 2023-03

VAReference baselines (pre-2024)Denoiser 9Mdiffusion

A variant implementing the same diffusion policy as the CNN variant with a Transformer denoiser.

Paper

Category: Reference baselines (pre-2024)
Type: VA
Released: 2023-03
Organization: Columbia University; Toyota Research Institute; MIT; Stanford University
Open weights: Yes
Parameters: Denoiser 9M (sim, including Block Push)·80M (Kitchen, real Push-T) + vision encoder 22M (Transport 45M, Block Push 0)

Backbone LLM: None
Vision encoder: ResNet-18
Action head: minGPT-style diffusion Transformer (8 layers) (diffusion)
Chunk: Predict 16 · execute 8
Control rate: 10 Hz
Architecture (original summary): ResNet-18 (no pretraining) + minGPT-style temporal diffusion Transformer (typically 8 layers, embedding 256 or 768; observation cross-attention, causal attention); action sequence Tp 16·Ta 8, real 10 Hz

Pretraining accelerators: N/A
Pretraining duration: N/A
Pretraining steps & batch: N/A
GPU/TPU-hours: Not disclosed
Fine-tuning: Per task: training GPU Not disclosed; same epoch budget as CNN variant (image 3000·state 4500), warmup 1000; real Push-T fixed 12 hours; (secondary) re-measured in CARP paper: V100 10.83 GPU-hours (state-based Square)
Data type: Simulation·synthetic; real-robot teleoperation
Data scale: Simulation same as the CNN variant; real Push-T 136 demos
Main datasets: Robomimic; Push-T; Block Push; Franka Kitchen; in-house real Push-T
Tasks & evaluation: Same sim set as the CNN variant; real Push-T
Robot: UR5 (real Push-T)

Inference device & speed: Paper: 0.1 s with 10 DDIM steps (RTX 3080, variant not specified); (secondary) VQ-BeT paper real-robot measurement DP-T 573.49 ms (RTX A4000)·5,243.82 ms (CPU)
Inference memory: Not disclosed
Weights bf16 (calc.): 0.2 GB
Where it runs: Local PC

Reinforcement learning: Not used · Imitation learning (diffusion behavior cloning)
Base control: Fixed · No base control (UR5 arm)

Key features: A variant implementing the same diffusion policy as the CNN variant with a Transformer denoiser. The sim denoiser is 9M, smaller than the CNN variant (256M).
Notes: Parameters (B) is the sum of real Push-T setting #D 80M + #V 22M (listed separately in the paper). Sim checkpoints released in the training log folder. Stanford added to affiliations (correction applied).
Verification: Corrections applied