NaVid

Peking University·BAAI·CASIA·University of Adelaide·Australian National University·Galbot · 2024-02

VLANavigation7Bdiscrete tokens

A video-VLM-based VLN agent that outputs the next action as language from monocular RGB video history and language instructions only, operating in continuous environments (VLN-CE) without maps·odometry·depth.

View architecture diagram in the gallery Paper ↗

Paper

NaVid: Video-based VLM Plans the Next Step for Vision-and-Language NavigationarXiv 2402.15852

Overview

Category: Navigation
Type: VLA
Released: 2024-02
Organization: Peking University·BAAI·CASIA·University of Adelaide·Australian National University·Galbot
Open weights: Yes
Parameters: 7B (based on Vicuna-7B; overall figure not stated)

Architecture

Backbone LLM: Vicuna-7B · LLaMA-VID config
Vision encoder: EVA-CLIP (EVA-ViT-G weights)
Action head: Next-action text (FORWARD+distance · TURN+angle · STOP, regex parsing) (discrete tokens)
Chunk: None (1 step)
Control rate: 1.2–1.5 s per action
Architecture (original summary): EVA-CLIP vision encoder + instruction-queried tokens · instruction-agnostic tokens (64 for the current frame, 4 per past frame) + Vicuna-7B (based on LLaMA-VID); outputs the next single action as text (FORWARD + distance, TURN-LEFT/RIGHT + angle, STOP), extracted by a regex parser; monocular RGB video input only, 1.2–1.5 s per action

Training

Pretraining accelerators: 24 × A100 (NaVid trained from LLaMA-VID weights)
Pretraining duration: About 28 h
Pretraining steps & batch: 1 epoch (batch not disclosed)
GPU/TPU-hours: 672 (Stated · A100-hours ("totaling 672 GPU hours"))
Fine-tuning: N/A (single co-training stage on navigation and web data)
Data type: Simulation·synthetic / web vision-language
Data scale: Navigation 510k samples (MP3D 61 scenes oracle steps 320k + non-oracle 180k + instruction-reasoning auxiliary 10k trajectories) + web data 763k (video captions, etc.)
Main datasets: VLN-CE R2R (61 MP3D scenes); web video-caption data 763k (detailed source not stated)
Tasks & evaluation: R2R-CE val-unseen SR 37.4%·SPL 35.9%·NE 5.47; RxR-CE cross-dataset (trained on R2R) SR 23.8%·SPL 21.2%; 200 real-robot trials with simple·complex instructions in 4 indoor scenes (Meeting room·Office·Lab·Lounge)
Robot: TurtleBot 4 wheeled + Azure Kinect DK (model uses RGB only; LiDAR·Nav2 for baseline models that need odometry)

Inference

Inference device & speed: Runs on 1 A100 on a remote server; the robot sends compressed images and receives parsed commands; about 1.2–1.5 s per action; memory not disclosed
Inference memory: Not disclosed
Weights bf16 (calc.): 14 GB
Where it runs: Remote server (A100)

Reinforcement learning & base

Reinforcement learning: Not used · Imitation learning (SFT) only; includes non-oracle trajectory data collection
Base control: Wheeled (Discrete motion actions) · VLM outputs discrete actions (forward distance·turn angle·stop) as text → robot executes commands and confirms turn·forward tracking

Notes

Key features: A video-VLM-based VLN agent that outputs the next action as language from monocular RGB video history and language instructions only, operating in continuous environments (VLN-CE) without maps·odometry·depth. Co-trained on 510k sim samples and 763k web data, it reached R2R-CE SR 37.4% and showed sim-to-real generalization in a real-robot test of 200 indoor instructions.
Notes: The README's released checkpoint (HF Jzzhang/NaVid, CC BY-NC 4.0) is a version retrained on R2R·RxR training splits following the Uni-NaVid training strategy, with R2R val SR 41.9%·RxR val SR 45.7% (README table), differing from paper numbers; evaluation code released; pretraining field refers to the NaVid training stage after LLaMA-VID initialization (only LLaMA·text encoder trained); batch·learning rate not disclosed; RSS 2024. Based on: LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models (arXiv 2311.17043) — the paper states it builds on the video VLM LLaMA-VID
Verification: Confirmed