VLA Gallery / All models / Flow·Diffusion VLA
HybridVLA
Peking University, BAAI, CUHK · 2025-03
VLAFlow·Diffusion VLATwo LLM sizes: 7B / 2.7Bhybrid
One LLM does both autoregressive prediction and diffusion action generation, using a collaborative ensemble that averages the two actions when mean AR token confidence exceeds 0.96.
Paper
HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action ModelarXiv 2503.10631
Overview
- Category
- Flow·Diffusion VLA
- Type
- VLA
- Released
- 2025-03
- Organization
- Peking University, BAAI, CUHK
- Open weights
- Yes
- Parameters
- Two LLM sizes: 7B / 2.7B; total not disclosed
Architecture
- Backbone LLM
- Llama 2 7B / Phi-2 2.7B
- Vision encoder
- SigLIP + DINOv2 (7B) / CLIP (2.7B)
- Action head
- Combined autoregressive tokens + diffusion tokens (4 steps) (hybrid)
- Chunk
- None (1 step)
- Control rate
- Not disclosed
- Architecture (original summary)
- Autoregressive tokens + diffusion tokens (<BOD>/<EOD>) within a single LLM (7B: LLaMA-2 + DINOv2·SigLIP, prism-dinosiglip-224px / 2.7B: Phi-2 + CLIP), 4 denoising steps (reduced from 20); single-step actions
Training
- Pretraining accelerators
- A800 (count not disclosed)
- Pretraining duration
- Not disclosed
- Pretraining steps & batch
- 5 epochs (35 datasets), batch not disclosed
- GPU/TPU-hours
- 10,000 (Stated · A800-hours (10K+))
- Fine-tuning
- RLBench: A800×8, 300 epochs, mixed precision (time not disclosed); README example A100×8 on 1 node, batch 32 per GPU
- Data type
- Real-robot teleoperation + simulation (RLBench fine-tuning)
- Data scale
- Pretraining 760K trajectories·33M frames, 35 datasets; RLBench 100 trajectories per task; real robot 100 demonstrations per task
- Main datasets
- Open X-Embodiment, DROID, RoboMIND
- Tasks & evaluation
- RLBench 10 tasks average 74% (7B); Franka FR3 real-robot 5 tasks average 0.80; AgileX bimanual 5 tasks average 0.66
- Robot
- Franka Research 3 (single arm), AgileX (bimanual); RLBench simulation
Inference
- Inference device & speed
- NVIDIA 4090D: HybridVLA(7B) 6.1 Hz, HybridVLA-dif(7B) 9.4 Hz, HybridVLA(2.7B) 12.3 Hz
- Inference memory
- Not disclosed
- Weights bf16 (calc.)
- 14 GB
- Where it runs
- Local PC
Reinforcement learning & base
- Reinforcement learning
- Not used · Imitation learning (behavior cloning) only
- Base control
- Fixed · No base control (Franka·AgileX bimanual manipulation·RLBench)
Notes
- Key features
- One LLM does both autoregressive prediction and diffusion action generation, using a collaborative ensemble that averages the two actions when mean AR token confidence exceeds 0.96. Pretrained on 760K trajectories from 35 datasets for over 10K A800 GPU-hours.
- Notes
- Parameters (B) is the maximum based on the LLM size of the paper variant HybridVLA (7B) (total including vision encoder not disclosed). Checkpoints released via Baidu Netdisk.
- Verification
- Confirmed