VLA Gallery / All models / Mobile manipulation

AC-DiT

Peking University·Nanjing University·CUHK·Beijing Academy of Artificial Intelligence(BAAI) · 2025-07

VLAMobile manipulationNot discloseddiffusion

Feeds the latent representation of a lightweight mobility head, pretrained on base actions alone, as a condition for whole-body action generation (mobility-to-body), and adjusts the weights of 2D images and 3D point clouds according to each stage's needs.

Paper

AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile ManipulationarXiv 2507.01961

Overview

Category

Mobile manipulation

Type

VLA

Released

2025-07

Organization

Peking University·Nanjing University·CUHK·Beijing Academy of Artificial Intelligence(BAAI)

Open weights

No

Parameters

Not disclosed (mobility head 170M; main DiT is the RDT-1B config per official code)

Architecture

Backbone LLM

None

Vision encoder

SigLIP so400m (wrist·chest RGB 3 views) + Lift3D-style point cloud tokenizer (LoRA)

Action head

Whole-body action diffusion DiT head (DDPM, 5 inference steps per code) (diffusion)

Chunk

2 steps (sim)

Control rate

Not disclosed (real demonstrations 30 fps)

Architecture (original summary)

SigLIP (so400m) image·text encoder + Lift3D-style point cloud tokenizer (LoRA) → weights 2D·3D tokens by language-vision cosine similarity → DiT action head (official code: RDT-1B config hidden 2048·28 layers, RDT-1B init); generates whole-body actions conditioned on latent base representation from pretrained mobility head (DiT 170M); base velocity (absolute) + arm joints (relative change in sim), chunk 2 (sim), DDPM inference 5 steps (code)

Training

Pretraining accelerators

N/A (official code initializes from external RDT-1B weights)

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

Not stated in paper (mobility head pretrained for 'a few epochs'). Official code (ManiSkill-HAB): mobility head 30,000 steps·batch 24 per GPU·lr 1e-4 → AC-DiT 20,000 steps·batch 16 per GPU·lr 1e-5·bf16; both scripts configured for 8 GPUs (model·time Not stated)

Data type

Simulation·synthetic / real-robot teleoperation

Data scale

ManiSkill-HAB 1,000 successful demos per task (generated by RL agents); RoboTwin 1,000 per task (motion planning); real robot 100 per task (30 fps)

Main datasets

ManiSkill-HAB generated demos; RoboTwin; in-house Cobot Magic demos

Tasks & evaluation

ManiSkill-HAB 7 subtasks average 55.6% (Table 1; 48.7% in text) vs RDT 42.9%·DP 28.8%·3DP 21.6%; RoboTwin 6 bimanual tasks average 90.1 (Table 2); 4 real long-horizon tasks (Cucumber in Basket·Store Breads·Hang Towel·Clean Table) 17 trials, best vs ACT·π0

Robot

Real Agilex Cobot Magic (4 Piper arms + Tracer wheeled base; 3 Orbbec Dabai on wrists·chest + head RealSense L515); sim ManiSkill-HAB (Fetch, per MS-HAB)·RoboTwin fixed bimanual

Inference

Inference device & speed

Not disclosed

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Partial (RL-generated data) · ManiSkill-HAB training demos generated by RL agents (open·close PPO, pick·place SAC); AC-DiT itself uses imitation learning

RL evidence

https://arxiv.org/abs/2507.01961

Base control

Wheeled (Direct base velocity/pose output) · Policy directly outputs base velocity (absolute) together with arm joint actions

Notes

Key features

Feeds the latent representation of a lightweight mobility head, pretrained on base actions alone, as a condition for whole-body action generation (mobility-to-body), and adjusts the weights of 2D images and 3D point clouds according to each stage's needs. Scored higher than RDT·DP·3DP on ManiSkill-HAB and beat ACT·π0 on 4 real Cobot Magic long-horizon tasks.

Notes

Paper does not state training·inference hardware or total parameters; RDT-1B initialization·steps·batch follow official GitHub scripts. ManiSkill-HAB average differs between table (55.6) and main text (48.7%). Code released 2026-02; trained AC-DiT weights not provided. NeurIPS 2025. Based on: RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (arXiv 2410.07864) — follows the RDT framework (cross-attention condition injection), and the official code initializes from RDT-1B weights

Verification

Confirmed