VLA Gallery / All models / Navigation

NWM

NWM (Navigation World Models)

Meta FAIR·New York University·UC Berkeley · 2024-12

기타Navigation1Bvideo world model

Trains a 1B conditional diffusion transformer (CDiT) world model on robot driving video and actions to imagine future video for action sequences, planning·scoring trajectories by similarity to the goal image.

Paper

Navigation World ModelsarXiv 2412.03572

Overview

Category

Navigation

Type

기타

Released

2024-12

Organization

Meta FAIR·New York University·UC Berkeley

Open weights

Yes

Parameters

1B (CDiT-XL; S 50M · B 200M · L 700M variants released)

Architecture

Backbone LLM

None

Vision encoder

Stable Diffusion VAE (latent encoding)

Action head

CEM planning / NoMaD sample ranking (goal-similarity energy) (video world model)

Chunk

8-step action sequence (CEM)

Control rate

Not disclosed (0.4 s per trajectory when distilled)

Architecture (original summary)

Stable Diffusion VAE latent + CDiT-XL (1B): 4 past frames of context, generates future frames with AdaLN conditioning on the sum of action (translation·yaw)·time shift k·diffusion step embeddings (250 denoising steps, 6 when distilled); planning: optimizes 8-step action sequences with CEM (N=120 candidates) or ranks NoMaD samples

Training

Pretraining accelerators

64 H100 (8 nodes × 8)

Pretraining duration

Not disclosed

Pretraining steps & batch

100k steps (HF card; 200k for the version with Ego4D) · final batch 4,096 (1,024 × 4 goals)

GPU/TPU-hours

Not disclosed

Fine-tuning

N/A

Data type

Real-robot teleoperation / real-robot autonomous driving logs / human video

Data scale

SCAND 8.7h·TartanDrive 5h·RECON 40h·HuRoN 75h+ + Ego4D 1,619 videos·908h+ (no action labels)

Main datasets

SCAND; TartanDrive; RECON; HuRoN(SACSoN); Ego4D (unlabeled); evaluation GO Stanford (unseen environments)

Tasks & evaluation

RECON planning ATE 1.13·RPE 0.35 (NoMaD 1.93·0.52); NoMaD samples ×32 ranking ATE 1.78·RPE 0.48 (×16 1.83·0.50); single-step·trajectory video prediction evaluated (LPIPS·DreamSim·FID)

Robot

No real-robot deployment (offline evaluation); training data are driving videos from multiple robots

Inference

Inference device & speed

On RTX 6000 Ada, one trajectory simulation 30.3 s → Time Skip 14.7 s → distillation (250→6 denoising) 0.4 s; 0.1 s with 4-bit quantization is an estimate; claims real-time use at 2–10 Hz

Inference memory

Not disclosed

Weights bf16 (calc.)

2 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Not used · Supervised diffusion world model + CEM planning (no RL)

Base control

Sim only (Motion set by planning/optimization) · No real-robot actuation: offline evaluation on recorded driving data by CEM-planning (translation·yaw) action sequences with the world model or ranking NoMaD samples

Notes

Key features

Trains a 1B conditional diffusion transformer (CDiT) world model on robot driving video and actions to imagine future video for action sequences, planning·scoring trajectories by similarity to the goal image. On RECON, standalone planning was more accurate (ATE 1.13·RPE 0.35) than NoMaD (1.93·0.52), and adding unlabeled Ego4D video improved prediction in unseen environments.

Notes

Training duration not disclosed. The HF released model was retrained on face-blurred data and may differ from paper figures (gated, CC BY-NC 4.0). Only offline planning evaluation, no real-robot runs. Total data hours not stated. CVPR 2025 Oral.

Verification

Confirmed