X-Mobility

NVIDIA·UT Austin·UC Berkeley · 2024-10

VANavigationNot disclosedregression

An end-to-end navigation model that first learns environment dynamics with an autoregressive latent world model, then separately trains an action policy on its latent state, generalizing from sim data alone.

View architecture diagram in the gallery Paper ↗

Paper

X-MOBILITY: End-To-End Generalizable Navigation via World ModelingarXiv 2410.17491

Overview

Category: Navigation
Type: VA
Released: 2024-10
Organization: NVIDIA·UT Austin·UC Berkeley
Open weights: Yes
Parameters: Not disclosed (total Not stated; components: image encoder 22.1M·state estimation 5.5M·prediction 2.3M·fusion 28.5M·action/path decoders 10.5M each·semantic decoder 34.2M·RGB diffuser 962M)

Architecture

Backbone LLM: None
Vision encoder: DINOv2 (22.1M, fine-tuned)
Action head: MLP action decoder: 6-dim velocity + 5-pose path (regression)
Chunk: None (1-step velocity; 5-pose path auxiliary)
Control rate: Not disclosed (Jetson P50 38.6 ms)
Architecture (original summary): DINOv2 image encoder (22.1M) + velocity encoder → GRU-based latent state estimation · prediction (autoregressive world model) → self-attention fusion with route embedding → MLP: 6-dim velocity command (linear · angular velocity) + 5 route poses; world model supervised by semantic · RGB (LDM) decoders during training

Training

Pretraining accelerators: 8 H100
Pretraining duration: Not disclosed
Pretraining steps & batch: world model 100 epochs (random action data) → joint policy training 100 epochs · batch 32
GPU/TPU-hours: Not disclosed
Fine-tuning: N/A (no real-robot fine-tuning)
Data type: Simulation·synthetic
Data scale: 4 Isaac Sim warehouses, recorded at 5 Hz: 100K Nav2 teacher frames + 160K random-action frames
Main datasets: x_mobility_isaac_sim_nav2_100k; x_mobility_isaac_sim_random_160k (HF nvidia/X-Mobility dataset)
Tasks & evaluation: Isaac Sim warehouse success rate 96% (BC 92%, Nav2 58%, MILE 34%); random obstacles 68% (BC 56%, Nav2 34%, MILE 26%); real Nova Carter single obstacle 10/10·multiple obstacles 8/9 (normal·dim lighting)
Robot: NVIDIA Nova Carter wheeled (real·sim), Forklift wheeled (Ackermann drive, sim); Unitree Go2 quadruped·G1 humanoid (sim, combined with RL locomotion policies)

Inference

Inference device & speed: Jetson AGX Orin: policy only P50 38.6 ms·P95 41.96 ms·GPU 594 MB; policy+semantic P50 55.55 ms·804 MB; supports ONNX·TensorRT conversion (README)
Inference memory: 594 MB (policy only) / 804 MB (policy + semantics)
Weights bf16 (calc.): Not disclosed
Where it runs: Onboard Jetson AGX Orin

Reinforcement learning & base

Reinforcement learning: Partial (RL low-level controller) · Only in sim Go2·G1 demos does an Isaac Lab RL locomotion policy convert velocity commands to joint control; X-Mobility itself is imitation-learned from a Nav2 teacher
RL evidence: https://arxiv.org/html/2410.17491
Base control: Wheeled·Quadruped·Humanoid legged (Direct base velocity/pose output) · Policy directly outputs linear and angular velocity commands (Nova Carter real and sim); for Go2 and G1 in sim, an Isaac Lab RL locomotion policy converts velocity commands into joint control

Notes

Key features: An end-to-end navigation model that first learns environment dynamics with an autoregressive latent world model, then separately trains an action policy on its latent state, generalizing from sim data alone. It reached 96% success in an Isaac Sim warehouse (Nav2 58%) and was deployed zero-shot on a real Nova Carter without fine-tuning.
Notes: 2-stage training (world model pretrained on random actions → policy added with Nav2 teacher data). Total parameters and training time not disclosed. HF checkpoint 1.44GB (x_mobility-nav2-semantic_action_path.ckpt), Apache 2.0 code. Base policy for the follow-up COMPASS.
Verification: Corrections applied