VQ-BeT

New York University; Seoul National University · 2024-03

VAVANot discloseddiscrete tokens

Converts continuous actions into hierarchical residual VQ codes and predicts them in one pass without iterative denoising.

Paper

Backbone LLM: None
Vision encoder: ResNet18 (frozen)
Action head: Residual VQ action codes + offsets (discrete tokens)
Chunk: Not disclosed
Control rate: Not disclosed
Architecture (original summary): minGPT (6 layers·6 heads·120-dim; BlockPush 4·4·72) + 2-layer residual VQ action codes (VQ-VAE latent 512, BlockPush 256; codebook 8–16) + offset, single-pass decoding; image tasks use frozen ResNet18 embeddings

Pretraining accelerators: N/A
Pretraining duration: N/A
Pretraining steps & batch: N/A
GPU/TPU-hours: Not disclosed
Fine-tuning: Per task: GPU·time·batch Not disclosed; epochs Kitchen 1000, Ant 300, BlockPush 1500, UR3 300, PushT 2000, NuScenes 1000, real 600; two-stage training (residual VQ tokenizer → transformer)
Data type: Simulation·synthetic; real-robot demonstrations (collection method not stated in note); other (nuScenes driving)
Data scale: PushT 206, Kitchen 566, BlockPush 1,000 trajectories, nuScenes 684 scenes; real 45 demos per task (80% for training)
Main datasets: Push-T; Franka Kitchen; BlockPush; UR3 BlockPush; Multimodal Ant; nuScenes
Tasks & evaluation: PushT, Image PushT, Kitchen, Image Kitchen, UR3 BlockPush, BlockPush, Multimodal Ant, nuScenes; real Stretch 5 single-stage·3 multi-stage·4 long-horizon tasks
Robot: Hello Robot Stretch (real); simulation; nuScenes driving

Inference device & speed: Real robot: VQ-BeT 18.06 ms (RTX A4000)·207.25 ms (4-core CPU) vs DP-T 573.49·5,243.82 ms
Inference memory: Not disclosed
Weights bf16 (calc.): Not disclosed
Where it runs: Local PC

Reinforcement learning: Not used · Imitation learning (behavior cloning) only
Base control: Fixed · No mention of base control (Stretch manipulation experiments; nuScenes is offline trajectory prediction)

Key features: Converts continuous actions into hierarchical residual VQ codes and predicts them in one pass without iterative denoising. Reported real-robot inference 25× faster than diffusion policy (5× in sim).
Notes: Real experiments use a Stretch mobile manipulator. Optional goal conditioning, no language. Correction applied to the structural values in the Table 13 BlockPush column. README provides only goal-conditioned Kitchen weights. ICML 2024.
Verification: Corrections applied