CARP

Westlake University; Zhejiang University; Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing · 2024-12†

VAVAState-based 0.65Mdiscrete tokens

Converts action sequences into multi-scale tokens and predicts them autoregressively from coarse to fine.

Paper

Category: VA
Type: VA
Released: 2024-12†
Organization: Westlake University; Zhejiang University; Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing
Open weights: Yes
Parameters: State-based 0.65M (DP-C 65.88M, DP-T 8.97M); image 7.58M; Kitchen 3.88M; multitask 16.08M

Backbone LLM: None
Vision encoder: Not stated
Action head: Multi-scale VQ tokens (coarse-to-fine) (discrete tokens)
Chunk: 16 steps
Control rate: Not disclosed
Architecture (original summary): Multi-scale action VQ-VAE tokenizer (one per action dimension, K=4 scales, 16-step horizon) + GPT-style transformer predicting discrete tokens from coarse to fine scales (cross-entropy)

Pretraining accelerators: N/A
Pretraining duration: N/A
Pretraining steps & batch: N/A
GPU/TPU-hours: Not disclosed
Fine-tuning: Multi-task: 1× A100, batch 512, 200 epochs; Square training cost (V100): tokenizer 4.67 GPU-hours (28 min×10) + policy 13.85 GPU-hours
Data type: Simulation·synthetic; real-robot teleoperation
Data scale: Robomimic 200 demos per task; MimicGen 1K trajectories per task; real 50 demos per task
Main datasets: Robomimic; Franka Kitchen; Push-T; MimicGen
Tasks & evaluation: Robomimic Lift·Can·Square (state·image), Kitchen, Push-T; MimicGen 8 tasks (multi-task); real UR5e 2 cup·bowl tasks
Robot: UR5e single arm; simulation

Inference device & speed: Measured as the average of 400 action predictions on A100, real robot uses RTX 2060; over 10× faster than DP
Inference memory: Not disclosed
Weights bf16 (calc.): 0 GB
Where it runs: Local PC

Reinforcement learning: Not used · Imitation learning (behavior cloning) only
Base control: Fixed · No base control (UR5e single arm·sim)

Key features: Converts action sequences into multi-scale tokens and predicts them autoregressively from coarse to fine. Reports inference more than 10× faster than diffusion policies.
Notes: Parameters (B) is the largest multitask 16.08M. Comparison parameters: image DP-C 255.61M, DP-T 9.01M; Kitchen DP-C 66.94M, DP-T 80.42M; multitask TCD 156.11M, SDP 159.85M (correction applied). In the same V100 measurement, DP-C 12.17 and DP-T 10.83 GPU-hours. Tasks specified by a learned 3-dim embedding (no language). v1 publication month could not be confirmed (†).
Verification: Corrections applied