VLingNav

ByteDance Seed·Peking University·Zhongguancun Academy · 2026-01

VLANavigation7Bregression

Navigation VLA with adaptive CoT that turns on explicit reasoning only when needed and a memory combining language summaries and key visual features; post-trained with expert-guided online RL after imitation learning.

View architecture diagram in the gallery Paper ↗

Paper

VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic MemoryarXiv 2601.08665

Overview

Category: Navigation
Type: VLA
Released: 2026-01
Organization: ByteDance Seed·Peking University·Zhongguancun Academy
Open weights: No
Parameters: 7B (based on LLaVA-Video-7B; overall figure not stated)

Architecture

Backbone LLM: Qwen2-7B · LLaVA-Video-7B config
Vision encoder: SigLIP-400M
Action head: MLP action head (Gaussian) waypoint trajectory (x, y, θ) + adaptive CoT text (regression)
Chunk: Not disclosed
Control rate: About 2.5 FPS (latency under 300 ms)
Architecture (original summary): SigLIP-400M + LLaVA-Video-7B; adaptive CoT (<think_on>/<think_off> instruction tokens, reasoning on only 2.1% of steps on average) + vision-assisted language memory (VLingMem, keeps past visual tokens·encodes only latest frame); last-token hidden state → MLP action head (Gaussian) outputs waypoint trajectory (x, y, θ)

Training

Pretraining accelerators: 128 × A100
Pretraining duration: Not disclosed
Pretraining steps & batch: 3 stages: open-ended adaptive CoT video pretraining 1 epoch → SFT co-training 20K steps · total batch 512 → online RL
GPU/TPU-hours: Not disclosed
Fine-tuning: Online RL: each iteration collects 128 on-policy episodes × 10 updates, PPO-style objective + REINFORCE++ advantage, expert-guided mixed objective (expert intervenes after 15 stalled steps); GPU·time not disclosed
Data type: Simulation·synthetic / web vision-language
Data scale: 4.5M total: navigation 2.9M (Nav-AdaCoT-2.9M, about 472K CoT annotations) + open-ended video 1.6M
Main datasets: Nav-AdaCoT-2.9M (HM3D·MP3D ObjectNav, tracking, ImageNav); LLaVA-Video-178K; Video-R1; ScanQA
Tasks & evaluation: ObjectNav SR HM3Dv1 79.1%·HM3Dv2 83.0%·MP3D 58.9%; HM3D-OVON val-unseen SR 50.1%; EVT-Bench SR single 88.4%·distracted 67.6%; HM3D Instance ImageNav SR 60.8%·SPL 37.4%; real-world Go2 ObjectNav·tracking·ImageNav in homes·offices·outdoors
Robot: Unitree Go2 quadruped + head-mounted RealSense D457 (RGB 1280×800, HFOV 90°)

Inference

Inference device & speed: Remote server RTX 4090; latency under 300 ms even at 500 frames, effective about 2.5 FPS; the robot tracks trajectories with NMPC
Inference memory: Not disclosed
Weights bf16 (calc.): 14 GB
Where it runs: Remote server (RTX 4090)

Reinforcement learning & base

Reinforcement learning: Used (VLA RL post-training) · Stage 3 expert-guided online RL (sim on-policy rollouts, PPO-style objective · REINFORCE++ advantage)
RL evidence: https://arxiv.org/abs/2601.08665
Base control: Quadruped (Waypoints → tracking controller) · VLA outputs waypoint trajectory → NMPC computes linear·angular velocity to track it

Notes

Key features: Navigation VLA with adaptive CoT that turns on explicit reasoning only when needed and a memory combining language summaries and key visual features; post-trained with expert-guided online RL after imitation learning. Reports HM3Dv2 ObjectNav SR 83.0% and state-of-the-art EVT-Bench and Instance ImageNav, transferring zero-shot to Unitree Go2.
Notes: Additional model: ByteDance Seed's 2026 navigation VLA, included for combining adaptive CoT, language memory, and online RL post-training and reporting state-of-the-art ObjectNav, tracking, and ImageNav; name of the LLM inside LLaVA-Video-7B not stated in the paper (the StreamVLN paper lists Qwen2-7B); number of waypoints n not stated; training duration not disclosed; weights and code not released (project page only); no VLN-CE results.
Verification: Confirmed