VLA GalleryAll modelsGallery

VLA Gallery / All models / Autoregressive & latent-action VLA

Moto

HKU·Tencent ARC Lab(PCG)·CUHK·UC Berkeley · 2024-12

VAAutoregressive & latent-action VLAGPT backbone 98Mlatent actions

Pretrains a GPT on latent motion tokens learned from video, then connects them to real actions.

View architecture diagram in the galleryPaper ↗

Paper

Moto: Latent Motion Token as the Bridging Language for Robot ManipulationarXiv 2412.04445

Overview

Category
Autoregressive & latent-action VLA
Type
VA
Released
2024-12
Organization
HKU·Tencent ARC Lab(PCG)·CUHK·UC Berkeley
Open weights
Yes
Parameters
GPT backbone 98M; tokenizer size not stated

Architecture

Backbone LLM
None
Vision encoder
VQ-VAE latent motion tokens
Action head
Latent motion tokens → actions (latent actions)
Chunk
Not disclosed
Control rate
Not disclosed
Architecture (original summary)
Moto-GPT (12 layers·768 hidden); VQ-VAE latent motion tokens (codebook 128, 8 tokens per frame pair); joint fine-tuning predicts motion tokens and actions together

Training

Pretraining accelerators
8 × '40G GPU' (Moto-GPT), 4 (tokenizer); GPU model not stated
Pretraining duration
Not disclosed
Pretraining steps & batch
Moto-GPT 10 epochs·batch 512; tokenizer 350k (OXE)/150k (CALVIN) steps·batch 256
GPU/TPU-hours
Not disclosed
Fine-tuning
4 × '40G GPU', 3 epochs (RT-1)/18 epochs (CALVIN); time not disclosed
Data type
Real-robot teleoperation (video) + human video
Data scale
Pre: OXE 109k trajectory videos + SSv2 105k human videos; fine-tune: RT-1 73k·CALVIN 18k trajectories, 90 real demonstrations
Main datasets
Open X-Embodiment, Something-Something V2, RT-1, CALVIN
Tasks & evaluation
CALVIN ABC→D average length 3.10; SIMPLER overall 0.614; real 60%
Robot
RT-1 robot (SIMPLER), CALVIN simulation, real robot (model not stated)

Inference

Inference device & speed
Not disclosed
Inference memory
Not disclosed
Weights bf16 (calc.)
0.2 GB
Where it runs
Not disclosed

Reinforcement learning & base

Reinforcement learning
Not used · Latent motion pretraining + imitation learning
Base control
Fixed · No base control (arm only: SIMPLER Google robot uses arm actions only)

Notes

Key features
Pretrains a GPT on latent motion tokens learned from video, then connects them to real actions. Uses only a 98M backbone without a pretrained VLM.
Notes
Classified as VA since it has no pretrained VLM. Parameters (B) are based on the 98M GPT backbone. Verification added the affiliations HKU·Tencent ARC·CUHK·UC Berkeley.
Verification
Corrections applied

Other models in this category