Uni-NaVid

Peking University·Galbot·BAAI · 2024-12

VLANavigation7Bdiscrete tokens

Unifies the input/output format of 4 navigation tasks (VLN·ObjectNav·EQA·human following) to train a single video VLA, achieving about 5 Hz non-blocking real deployment with online token merging and 4-step action prediction.

View architecture diagram in the gallery Paper ↗

Paper

Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation TasksarXiv 2412.06224

Overview

Category: Navigation
Type: VLA
Released: 2024-12
Organization: Peking University·Galbot·BAAI
Open weights: Yes
Parameters: 7B (based on Vicuna-7B; overall figure not stated)

Architecture

Backbone LLM: Vicuna-7B
Vision encoder: EVA-CLIP
Action head: 4 discrete action tokens (forward 25 cm · turn left/right 30° · stop) or text answer (discrete tokens)
Chunk: 4 actions
Control rate: About 5 Hz (inference 0.2 s)
Architecture (original summary): EVA-CLIP + 2-layer MLP projector + Vicuna-7B (LLaMA-VID · NaVid family); online visual token merging (64 current · 4 short-term memory · 1 long-term memory per frame group); next 4 discrete action tokens (FORWARD 25 cm · TURN-LEFT/RIGHT 30° · STOP) or text answer (EQA); about 5 Hz on a remote A100

Training

Pretraining accelerators: 40 H800
Pretraining duration: About 35 h
Pretraining steps & batch: 1 epoch (batch not disclosed)
GPU/TPU-hours: 1,400 (Stated · H800-hours ("totaling 1400 GPU hours"))
Fine-tuning: N/A (two-stage training: stage 1 aligns only the projector, stage 2 co-trains navigation and video QA; README provides additional fine-tuning scripts based on Uni-NaVid)
Data type: Simulation·synthetic / web vision-language
Data scale: Navigation 3.6M samples (VLN 2.4M, ObjectNav 483k, EQA video-action 240k + video-answer 10k, person following 544k) + video QA 2.3M
Main datasets: VLN-CE R2R·RxR; HM3D ObjectNav; MP3D-EQA; Habitat 3.0 human following (in-house collection); public video QA 2.3M
Tasks & evaluation: R2R-CE val-unseen SR 47.0%·SPL 42.7%; RxR-CE SR 48.7%·SPL 40.9%; HM3D ObjectNav SR 73.7%·SPL 37.1%; MP3D-EQA ACC 47.3% (54.4% with GT trajectories); person following (in-house Habitat 3.0 benchmark) SR 61.21%; real-world complex instruction demos
Robot: Unitree Go2 quadruped + head-mounted RealSense D455 (model input is monocular RGB; built-in LiDAR-L1 only for local motion planning)

Inference

Inference device & speed: Inference about 0.2 s (about 5 Hz) on 1 A100 on a remote server + internet communication (image upload·command receipt) about 0.3 s; README mentions further speedup possible with quantization; memory not disclosed
Inference memory: Not disclosed
Weights bf16 (calc.): 14 GB
Where it runs: Remote server (A100)

Reinforcement learning & base

Reinforcement learning: Not used · Imitation learning (SFT) only
Base control: Quadruped (Discrete motion actions) · VLA outputs 4 discrete actions (forward 25 cm·turn 30°·stop) → robot executes them sequentially in non-blocking fashion (LiDAR for local motion planning)

Notes

Key features: Unifies the input/output format of 4 navigation tasks (VLN·ObjectNav·EQA·human following) to train a single video VLA, achieving about 5 Hz non-blocking real deployment with online token merging and 4-step action prediction. Through cross-task synergy on 3.6M navigation samples, it reported state of the art at the time on multiple benchmarks, including R2R-CE SR 47.0% and HM3D ObjectNav SR 73.7%.
Notes: Public weights (HF Jzzhang/Uni-NaVid, uninavid-7b-full-224-video-fps-1-grid-2) are an open-source version with some features rewritten due to licensing and do not use L3MVN data for ObjectNav; README table R2R val SR 51.8%·RxR val SR 56.1% differ from paper figures; training·evaluation code released; batch·learning rate not disclosed; RSS 2025. Based on: NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation (arXiv 2402.15852) — README: code is based on LLaMA-VID·NaVid
Verification: Corrections applied