VLA Gallery / All models / Flow·Diffusion VLA
CogACT
Tsinghua University, Microsoft Research Asia, USTC, Institute of Microelectronics CAS · 2024-11
VLAFlow·Diffusion VLA7.6Bdiffusion
A separate DiT generates action sequences conditioned on the cognition token from the VLM, and Adaptive Action Ensemble merges chunks.
Paper
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic ManipulationarXiv 2411.19650
Overview
- Category
- Flow·Diffusion VLA
- Type
- VLA
- Released
- 2024-11
- Organization
- Tsinghua University, Microsoft Research Asia, USTC, Institute of Microelectronics CAS
- Open weights
- Yes
- Parameters
- 7.6B (body, value compared with RT-2-X 55B); action DiT Small 13M/Base 89M (default)/Large 308M (supplementary table) · up to 300M (body)
Architecture
- Backbone LLM
- Llama 2 7B · Prismatic-7B
- Vision encoder
- SigLIP + DINOv2
- Action head
- DiT diffusion action module (Base 89M, DDIM 10) (diffusion)
- Chunk
- 16 steps
- Control rate
- Not disclosed
- Architecture (original summary)
- Prismatic VLM (DINOv2+SigLIP, LLaMA-2) cognition token + DiT diffusion action module (Small 13M/Base 89M/Large 308M); 16-step×7-DoF chunks (future N=15), 8 diffusion steps per sample during training, inference DDIM 10 steps·cfg 1.5
Training
- Pretraining accelerators
- A100×16(FSDP)
- Pretraining duration
- About 5 days
- Pretraining steps & batch
- 135K+ iter, batch 256, constant lr 2e-5
- GPU/TPU-hours
- 1,920 (Converted (approx.) · A100-hours (16 × about 5 days))
- Fine-tuning
- Real robot (Realman): full fine-tuning with FSDP on 16×A100, checkpoint at 10K steps, 7.5 h; README: 8×A100 1 node, global batch 256 (32 per device), about 30 epochs recommended
- Data type
- Real-robot teleoperation
- Data scale
- 25-dataset OXE subset, 0.4M trajectories·22.5M frames; real-robot fine-tuning Realman 391 demos, Franka 100 per task (400 total)
- Main datasets
- Open X-Embodiment subset (25)
- Tasks & evaluation
- SIMPLER Google Robot VM 74.8%·VA 61.3%, WidowX VM 51.3%; Realman real-robot average 71.2% (Pick 70.8, Stack 82.3, Place 60.4); Franka real-robot 4-task average 61.4%
- Robot
- Google Robot, WidowX, Realman, Franka (single arm)
Inference
- Inference device & speed
- A6000 bf16: about 181 ms per call (16 actions), about 5.5 Hz with Adaptive Action Ensemble (OpenVLA on the same hardware 307 ms · 1 action); fp32 about 30 GB
- Inference memory
- fp32 about 30 GB (A6000)
- Weights bf16 (calc.)
- 15.2 GB
- Where it runs
- Local PC
Reinforcement learning & base
- Reinforcement learning
- Not used · Imitation learning (behavior cloning) only
- Base control
- Fixed · No base control (single 7-DoF arm)
Notes
- Key features
- A separate DiT generates action sequences conditioned on the cognition token from the VLM, and Adaptive Action Ensemble merges chunks. At 7.6B it achieved SIMPLER Google Robot VM 74.8% and a Realman real-robot average of 71.2%.
- Notes
- The original does not state which DiT size the 7.6B refers to (default action module is DiT-Base). The note's 'Franka 30K checkpoint' is a baseline (Octo · OpenVLA) value and was removed; CogACT tested only a single 10K-step checkpoint. Largest DiT is 308M in the table/300M in the body. Checkpoint 30 GB.
- Verification
- Corrections applied