VLA Gallery / All models / Navigation

StreamVLN

Shanghai AI Laboratory·The University of Hong Kong·Zhejiang University·Shanghai Jiao Tong University · 2025-07

VLANavigation8.03Bdiscrete tokens

A streaming VLN model that processes continuous video as a multi-turn dialogue, reusing recent turns via a sliding-window KV cache and compressing past visual information into a 3D voxel-based token-pruning memory.

Paper

StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context ModelingarXiv 2507.05240

Overview

Category

Navigation

Type

VLA

Released

2025-07

Organization

Shanghai AI Laboratory·The University of Hong Kong·Zhejiang University·Shanghai Jiao Tong University

Open weights

Yes

Parameters

8.03B (HF safetensors; based on LLaVA-Video 7B)

Architecture

Backbone LLM

Qwen2-7B · LLaVA-Video 7B config

Vision encoder

LLaVA-Video built-in vision encoder (name not stated)

Action head

Discrete action text (↑ 25 cm · ←/→ 15° · STOP), 4 per turn (discrete tokens)

Chunk

4 actions (code default)

Control rate

0.27 s per 4 actions (RTX 4090)

Architecture (original summary)

Multi-turn conversational streaming based on LLaVA-Video 7B (LLM Qwen2-7B); fast: sliding-window KV cache reuse of the last N turns, slow: 3D voxel-based token pruning memory; outputs discrete action text each turn (↑ forward 25 cm·←/→ turn 15°·STOP, 4 by default in code)

Training

Pretraining accelerators

A100 (count not disclosed)

Pretraining duration

Not disclosed

Pretraining steps & batch

2 stages: oracle VLN trajectories 1 epoch → 1 more epoch of DAgger · VL data co-training; 128 video clips per step (max lr LLM 2e-5 · vision encoder 5e-6)

GPU/TPU-hours

1,500 (Stated · A100-hours ("around 1500 A100 GPU hours"))

Fine-tuning

N/A (separate checkpoint released for real-world deployment, training compute not disclosed)

Data type

Simulation·synthetic / web vision-language

Data scale

VLN video clips 450K (60 MP3D environments, R2R·R2R-EnvDrop·RxR) + ScaleVLN subset 300K (150k in another section) + DAgger 240K + video VQA 248K + MMC4 interleaved 230K; mixing ratio VLA 67%·general 33%

Main datasets

R2R-CE; RxR-CE; R2R-EnvDrop; ScaleVLN (HM3D, VLN-CE-converted subset); LLaVA-Video-178K; ScanQA·SQA3D; MMC4

Tasks & evaluation

R2R-CE val-unseen SR 52.8%·SPL 47.2% (extra VLN data † SR 56.4%·SPL 50.2%); RxR-CE SR 48.6%·SPL 42.5% († SR 54.4%·SPL 45.4%); real-robot Go2 executes simple·long-horizon instructions indoors·outdoors

Robot

Unitree Go2 quadruped + RealSense D455 (model input RGB; depth·odometry for token-pruning back-projection)

Inference

Inference device & speed

Remote workstation RTX 4090; inference for 4 actions averages 0.27 s + communication 0.2 s (indoor)·1.0 s (outdoor); Go2 streams video to the server; memory not disclosed

Inference memory

Not disclosed

Weights bf16 (calc.)

16.1 GB

Where it runs

Remote workstation (RTX 4090)

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning (SFT) + co-training on DAgger-collected data

Base control

Quadruped (Discrete motion actions) · VLA outputs a sequence of discrete actions (forward 25 cm·turn 15°·stop) → robot client publishes velocity commands via ROS2

Notes

Key features

A streaming VLN model that processes continuous video as a multi-turn dialogue, reusing recent turns via a sliding-window KV cache and compressing past visual information into a 3D voxel-based token-pruning memory. It keeps latency low on long videos, reached R2R-CE SR 56.4% (extra VLN data), and executed indoor·outdoor instructions on a real Unitree Go2.

Notes

Based on arXiv v2 (2026-07); body figures (R2R SR 56.9%·SPL 51.9%, RxR SR 52.9%·SPL 46.0%) and Table I (56.4/50.2, 54.4/45.4) disagree; inconsistent description of action token count ('8 action tokens' vs '0.27s for 4 actions'), official code default num_future_steps=4; ScaleVLN 300K vs 150k inconsistent; GPU count·duration not disclosed; code·data·2 checkpoints (benchmark·real-world) released, CC BY-NC-SA 4.0. Based on: LLaVA-Video: Video Instruction Tuning With Synthetic Data (arXiv 2410.02713) — paper·README state it is built on LLaVA-Video 7B

Verification

Confirmed