NoMaD

UC Berkeley · 2023-10

VANavigation19Mdiffusion

Handles goal-directed navigation and goal-free exploration in one diffusion policy via goal masking, which randomly masks the goal image.

Paper

Backbone LLM: None
Vision encoder: EfficientNet-B0 (observation / observation + goal)
Action head: 1D conditional U-Net diffusion head (K=10) + temporal distance head (diffusion)
Chunk: 8 waypoints (public config)
Control rate: Not disclosed (deployment config 4 Hz)
Architecture (original summary): EfficientNet-B0 tokenizes observation · goal into 256-dim tokens → Transformer 4 layers · 4 heads (goal masking p_m=0.5) → 1D conditional U-Net diffusion head (K=10 denoising) generates future action sequence (8 waypoints in public config) + temporal distance head; PD controller converts to velocity

Pretraining accelerators: Not disclosed (acknowledgments mention only compute support such as Google TPU Research Cloud)
Pretraining duration: Not disclosed
Pretraining steps & batch: 30 epochs · batch 256 (public nomad.yaml uses 100 epochs)
GPU/TPU-hours: Not disclosed
Fine-tuning: N/A
Data type: Real-robot teleoperation / real-robot autonomous driving logs
Data scale: Over 100 hours of GNM · SACSoN datasets
Main datasets: GNM data mix (RECON, GoStanford, SCAND, TartanDrive, etc.); SACSoN (HuRoN)
Tasks & evaluation: Real-world exploration success rate 98%·0.2 collisions (Subgoal Diffusion 77%·1.7, Random Subgoals 70%); goal-conditioned navigation 90%; vs specialist policies exploration 98% (Diffusion Policy 98%)·goal 92% (ViNT 92%)
Robot: LoCoBot wheeled (indoor·outdoor environments)

Inference device & speed: Aimed at onboard execution on Jetson Orin class (paper); README deployment example LoCoBot + Jetson Orin Nano; 10 diffusion steps; latency not disclosed (deployment loop set to 4 Hz)
Inference memory: Not disclosed
Weights bf16 (calc.): 0 GB
Where it runs: Onboard Jetson Orin-class (README: Orin Nano)

Reinforcement learning: Not used · End-to-end supervised imitation learning only
Base control: Wheeled (Waypoints → tracking controller) · Diffusion policy outputs a waypoint sequence → PD controller converts it to base velocity commands (README)

Key features: Handles goal-directed navigation and goal-free exploration in one diffusion policy via goal masking, which randomly masks the goal image. With 19M parameters it improved exploration success (98% vs 77%) and collisions (0.2 vs 1.7) over a 335M subgoal-diffusion approach.
Notes: Paper does not state action sequence length·context·resolution → per public nomad.yaml: 8 waypoints·context 3·96×96. Training GPU·time Not disclosed. Combines goal masking with the ViNT policy architecture. Weights released on Google Drive. ICRA 2024. Based on: ViNT: A Foundation Model for Visual Navigation (arXiv 2306.14846) — combines goal masking·diffusion head with the ViNT policy architecture (stated in paper)
Verification: Confirmed