NavDP

Shanghai AI Laboratory·Tsinghua University·Zhejiang University·The University of Hong Kong · 2025-05

VANavigationNot discloseddiffusion

Using only simulation data, jointly trains trajectory generation (diffusion) and trajectory evaluation (critic) from RGB-D observations in one transformer; the critic is trained on contrastive negative trajectories built from sim privileged information (global ESDF).

View architecture diagram in the gallery Paper ↗

Paper

NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information GuidancearXiv 2505.08712

Overview

Category: Navigation
Type: VA
Released: 2025-05
Organization: Shanghai AI Laboratory·Tsinghua University·Zhejiang University·The University of Hong Kong
Open weights: Yes
Parameters: Not disclosed

Architecture

Backbone LLM: None
Vision encoder: DepthAnything ViT (RGB, pretrained) + ViT from scratch (depth)
Action head: Waypoint diffusion head (conditional U-Net·DDPM) + critic selection (diffusion)
Chunk: 24 waypoints
Control rate: >10 Hz
Architecture (original summary): RGB encoded by pretrained DepthAnything ViT, depth by a from-scratch ViT → Transformer decoder compresses 512 RGB-D tokens into 16 fused tokens → 2-layer Transformer encoder (shared by policy·critic) → conditional U-Net·DDPM diffusion head generates a batch of 24-step waypoints (Δx,Δy,Δω), critic (trajectories encoded by 1D conv) scores them to select the best trajectory; linear·angular velocity computed from trajectory midpoint×proportional gain, >10 Hz

Training

Pretraining accelerators: Not disclosed
Pretraining duration: Not disclosed
Pretraining steps & batch: Not disclosed
GPU/TPU-hours: Not disclosed
Fine-tuning: Additional training on real-to-sim data (3DGS lab reconstruction, about 4k trajectories): method·accelerator·time not disclosed
Data type: Simulation·synthetic
Data scale: 56,000+ trajectories·10M RGB-D images·363.2 km·1,244 scenes (main text; arXiv abstract metadata says '1M+ m·3,000 scenes'); about 2,500 trajectories generated per GPU per day; real-to-sim about 4k trajectories
Main datasets: In-house sim-generated data (3D-Front 1,176 scenes, Matterport3D 68 scenes); 3DGS lab reconstruction (real-to-sim)
Tasks & evaluation: In-house IsaacSim benchmark (Dingo, Go2, R1) PointNav sim SR 81.3/83.0/75.0%, mean over 6 sim and real-to-sim settings mSR 70.4, mSPL 58.6 (ViPlanner 65.6, 55.4); goal-free exploration, ImageNav; real-robot zero-shot, real-world success 50%→80% when adding in-domain real-to-sim data; sim goal-free exploration area Dingo 280.2, Go2 359.1, R1 300.4 (NoMaD 119.2, 36.9, 57.9; ViNT 38.4, 37.7, 61.3)
Robot: Real robots Unitree Go2 quadruped, Galaxea R1, Unitree G1 humanoid; sim ClearPath Dingo (2-wheel differential)·Go2·R1

Inference

Inference device & speed: Real-time inference at >10 Hz on a GeForce RTX 5080 laptop GPU; robot up to 2.0 m/s; latency (ms) and memory not disclosed
Inference memory: Not disclosed
Weights bf16 (calc.): Not disclosed
Where it runs: RTX 5080 laptop GPU

Reinforcement learning & base

Reinforcement learning: Partial (RL low-level controller) · NavDP itself uses imitation learning (diffusion) + supervised critic learning with sim privileged information (global ESDF); sim Go2 quadruped tracks trajectories with an RL-trained locomotion policy
RL evidence: https://arxiv.org/html/2505.08712v3
Base control: Wheeled·Quadruped·Humanoid legged (Waypoints → tracking controller) · Policy outputs 24-step waypoints → converted to linear and angular velocity via trajectory midpoint × proportional gain for base commands

Notes

Key features: Using only simulation data, jointly trains trajectory generation (diffusion) and trajectory evaluation (critic) from RGB-D observations in one transformer; the critic is trained on contrastive negative trajectories built from sim privileged information (global ESDF). Without real-robot data it transfers zero-shot to quadruped·wheeled·humanoid robots, and adding Gaussian Splatting real-to-sim data (about 4k trajectories) raises real-world success from 50% to 80%.
Notes: Training GPU, time, steps, batch, and parameter count all not disclosed (text and appendix). Conflicting data scale: text, PDF, and HF abstract say 363.2 km and 1,244 scenes, while current arXiv abstract metadata says 'over 1 million m and 3,000 scenes'. Checkpoints provided after a Google Form request (InternVLA-N1 System-1 model per README), code CC BY-NC-SA 4.0. The RL post-trained version X-NavDP (2607.28560) is a separate paper.
Verification: Corrections applied