VLA Gallery / All models / Navigation

NaVILA

UC San Diego·USC·NVIDIA · 2024-12

VLANavigation8Bdiscrete tokens

A two-stage design where the VLA outputs mid-level language actions with distances·angles instead of low-level joint commands, and an RL locomotion policy using a LiDAR height map executes them with obstacle avoidance.

Paper

NaVILA: Legged Robot Vision-Language-Action Model for NavigationarXiv 2412.04453

Overview

Category

Navigation

Type

VLA

Released

2024-12

Organization

UC San Diego·USC·NVIDIA

Open weights

Yes

Parameters

8B (NaVILA-8B)

Architecture

Backbone LLM

Llama 3 8B · VILA config (README checkpoint name)

Vision encoder

SigLIP (README checkpoint name)

Action head

Intermediate language action text (forward cm · rotation degrees · stop) → fixed velocity commands (discrete tokens)

Chunk

None (1 step)

Control rate

VLA about 1 FPS (RTX 4090)

Architecture (original summary)

VILA-based 8B VLM (README checkpoint name: SigLIP vision encoder + Llama 3 8B) + MLP projector, 196 tokens per frame·8-frame memory in real deployment; mid-level language actions ('move forward 75 cm', 'turn right 30 degrees', 'stop') → executed at fixed velocity (0.5 m/s, ±π/6 rad/s, 0) for the corresponding duration → RL locomotion policy (LiDAR height map + proprioception) outputs joint position targets

Training

Pretraining accelerators

4 A100 nodes (VLA instruction-tuning stage; stages 1 · 2 inherited from VILA used 16 A100 nodes, 8 GPUs per node)

Pretraining duration

18 hours (instruction-tuning stage)

Pretraining steps & batch

1 epoch (lr 1e-4 cosine decay · warm-up 0.03; batch not disclosed)

GPU/TPU-hours

Not disclosed

Fine-tuning

No separate VLA fine-tuning; RL locomotion policy via Isaac Lab PPO, 60K+ FPS throughput on 1 RTX 4090 (training time not disclosed)

Data type

Simulation·synthetic / web video / web vision-language

Data scale

2K YouTube egocentric tour videos → 20K trajectories (action labels via MASt3R pose estimation); scale of R2R·RxR·EnvDrop·ScanQA·general VQA not stated

Main datasets

R2R-CE; RxR-CE; R2R-EnvDrop; ScanQA; YouTube Human Touring (in-house); general video VQA (video_chatgpt·sharegpt_video·sharegpt4v_sft, README)

Tasks & evaluation

R2R-CE val-unseen SR 54.0%·SPL 49.0%; RxR-CE SR 49.3%·SPL 44.0%; introduces in-house VLN-CE-Isaac benchmark (R2R val-unseen 1,077 trajectories); real-world Workspace·Home·Outdoor 25 instructions×3 trials SR 88%, complex instructions 75%

Robot

Unitree Go2 quadruped (LiDAR); Unitree H1·Booster T1 humanoids

Inference

Inference device & speed

VLA about 1 FPS on 1 RTX 4090; FP16 594.58 ms·18.5 GB, AWQ W4A16 367.80 ms·8.6 GB (1,737 context tokens); execution location not specified (notes that quantization could remove image transfer time when deployed directly on the robot)

Inference memory

18.5 GB (FP16)

Weights bf16 (calc.)

16 GB

Where it runs

Single RTX 4090 (onboard or not unspecified)

Reinforcement learning & base

Reinforcement learning

Partial (RL low-level controller) · VLA by SFT; low-level locomotion policy trained with PPO in Isaac Lab (LiDAR height map, joint position output)

RL evidence

https://arxiv.org/abs/2412.04453

Base control

Quadruped·Humanoid legged (Commands/latents → RL locomotion controller) · VLA outputs mid-level language commands (distance·angle·stop) → converted to fixed velocity commands → RL locomotion policy controls joint positions

Notes

Key features

A two-stage design where the VLA outputs mid-level language actions with distances·angles instead of low-level joint commands, and an RL locomotion policy using a LiDAR height map executes them with obstacle avoidance. Trained on data including 20K trajectories built from 2K YouTube tour videos, it reaches R2R-CE SR 54.0% and 88% success on 25 real-world instructions.

Notes

LLM·vision encoder names not stated in the paper; based on README checkpoint names (navila-siglip-llama3-8b-v1.5-pretrain, navila-llama3-8b-8f); VILA stages 1·2 (connector initialization 4 hours·VL pretraining 30 hours) inherited; RL locomotion policy control frequency not stated (LiDAR 15 Hz); weights·training code·data annotations released (YouTube originals as video IDs only); RSS 2025. Based on: VILA: On Pre-training for Visual Language Models (arXiv 2312.07533) — the paper states it builds on VILA and inherits stages 1·2 from VILA

Verification

Confirmed