CrossFormer

UC Berkeley·Carnegie Mellon University · 2024-08

VLANavigation130Mregression

Trains a single 130M Transformer on 900K trajectories from 20 embodiments without aligning observation or action spaces, controlling single and bimanual arms, wheeled robots, quadrupeds and quadcopters.

View architecture diagram in the gallery Paper ↗

Paper

Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and AviationarXiv 2408.11812

Overview

Category: Navigation
Type: VLA
Released: 2024-08
Organization: UC Berkeley·Carnegie Mellon University
Open weights: Yes
Parameters: 130M

Architecture

Backbone LLM: None
Vision encoder: ResNet-26 (ImageNet init, FiLM language conditioning)
Action head: Per-embodiment L1 regression head (4 navigation waypoints·12-D quadruped joints etc.) (regression)
Chunk: 4 nav waypoints (arm 4·bimanual 100·quadruped 1)
Control rate: Nav 4 Hz · quadruped 20 Hz · bimanual 20 Hz · arm 5–15 Hz
Architecture (original summary): ResNet-26 image tokenizer (language FiLM conditioning) + proprioception linear projection → causal Transformer 12 layers·8 heads (512-dim, per-embodiment readout tokens) → L1 regression head: navigation 4 2D waypoints (4 Hz), single-arm EEF 4 steps (5–15 Hz), bimanual joints 100 steps (20 Hz), quadruped 12-dim joint positions 1 step (20 Hz)

Training

Pretraining accelerators: TPU v5e-256 pod
Pretraining duration: 47 hours (Sec. 3.4 · README)/80 hours (Appendix B)
Pretraining steps & batch: 300k steps · batch 512
GPU/TPU-hours: 12,032 (Converted · TPU v5e 256 chips × 47 hours (20,480 if Appendix B's 80 hours applied))
Fine-tuning: N/A (evaluation uses the pretrained model as is; README provides fine-tuning scripts)
Data type: Real-robot teleoperation / real-robot autonomous driving logs
Data scale: 900K trajectories · 20 embodiments (30 in Figure 1); GNM navigation 60 hours (17% sampling share), ALOHA 7K trajectories, Go1 locomotion 25 minutes
Main datasets: Open X-Embodiment (Octo single-arm mix); DROID; ALOHA-multi-task; GNM; Go1-walk; Franka-tabletop
Tasks & evaluation: Average success over 6 robots 73% (single-robot policies 67%, previous best 51%); LoCoBot obstacle avoidance 0.95 · corner 0.95 · sharp corner 0.88 (previous best method 0.30 · 0.85 · 0.30); Tello corner 0.82 (baseline 0.68)
Robot: WidowX·Franka single arm, ALOHA bimanual (fixed); LoCoBot wheeled; Unitree Go1 quadruped; DJI Tello quadcopter (no aerial training data)

Inference

Inference device & speed: Nav·drone: inference on a separate server then wireless transmission; manipulation: local RTX 4090-class GPU; nav 4 Hz·quadruped 20 Hz; latency Not disclosed (inference speed cited as a limitation)
Inference memory: Not disclosed
Weights bf16 (calc.): 0.3 GB
Where it runs: Remote server (navigation·drone, wireless transmission)

Reinforcement learning & base

Reinforcement learning: Partial (RL-generated data) · Only Go1 locomotion data collected via sim RL expert policy rollouts; CrossFormer itself uses L1 imitation learning
RL evidence: https://arxiv.org/pdf/2408.11812
Base control: Wheeled·Quadruped·Aerial (Waypoints → tracking controller) · Nav head outputs 4 relative 2D waypoints (LoCoBot follows topological-graph subgoals, Tello at fixed altitude) / quadruped head directly outputs 12-dim leg joint positions at 20 Hz / arm head outputs EEF·joint targets

Notes

Key features: Trains a single 130M Transformer on 900K trajectories from 20 embodiments without aligning observation or action spaces, controlling single and bimanual arms, wheeled robots, quadrupeds and quadcopters. Average success over 6 robots was 73%, higher than single-robot policies (67%) and the previous best cross-embodiment method (51%).
Notes: Training time inconsistent: 47h in the text and README vs 80h in Appendix B. Number of embodiments inconsistent: 20 in the text vs 30 in Figure 1. Go1 data are rollouts of sim RL expert policies. Language encoder not specified in the paper (official config uses Universal Sentence Encoder). HF checkpoint 519MB (300k steps), MIT license. LoCoBot navigation is conditioned on a topological graph and goal images. Total data hours not stated. Based on: Octo: An Open-Source Generalist Robot Policy (arXiv 2405.12213) — based on the Octo codebase (stated in README)
Verification: Confirmed