VLA Gallery / All models / Navigation

ViNT

UC Berkeley · 2023-06

VANavigation31Mregression

Trains a 31M Transformer policy on multi-robot driving data that extends the GNM data mix, and adapts by replacing the goal token with soft prompts for other goal modalities such as GPS coordinates and route commands.

Paper

ViNT: A Foundation Model for Visual NavigationarXiv 2306.14846

Overview

Category

Navigation

Type

VA

Released

2023-06

Organization

UC Berkeley

Open weights

Yes

Parameters

31M (policy; diffusion subgoal model 318M separate)

Architecture

Backbone LLM

None

Vision encoder

EfficientNet-B0 (observation / observation + goal early fusion)

Action head

5 waypoints + temporal distance MLP regression (regression)

Chunk

5 waypoints

Control rate

4 Hz

Architecture (original summary)

EfficientNet-B0 observation encoder + goal encoder concatenating observation · goal channels (early fusion) → decoder-style Transformer 4 layers · 4 heads (512-dim, 5 past + current observation tokens + goal token) → MLP: 5 relative waypoints + temporal distance; 85×64 RGB; runs at 4 Hz · PD tracking; during exploration a 318M image diffusion model proposes subgoals + topological graph

Training

Pretraining accelerators

8 V100 (Table 6, representative configuration); diffusion model on TPU v4-8 board

Pretraining duration

30 hours (policy); diffusion model 30 hours

Pretraining steps & batch

30 epochs · batch 300 (policy); diffusion 250k steps · batch 128

GPU/TPU-hours

240 (Converted · 8× V100 × 30 hours (Table 6 'representative configuration' footnote; diffusion model TPU v4-8 × 30 hours separately))

Fine-tuning

Adaptation to new environments·embodiments·goal modalities (GPS·route commands): lr 1e-4·5 epochs, around 1 h of data; GPU·time not disclosed

Data type

Real-robot teleoperation / real-robot autonomous driving logs

Data scale

Table 7: 80h used for training out of 160h across 11 datasets; 8 robot platforms (text)

Main datasets

RECON; GoStanford; SCAND; TartanDrive; NeBula; SACSoN; BDD; CoryHall; Berkeley; Seattle

Tasks & evaluation

Undirected goal-reaching success indoor 0.94·outdoor 1.00 (End-to-End BC 0.72·0.44); satellite-guided exploration success 1.00·SPL 0.94 (ViKiNG 0.77·0.68); adaptation success image 0.82·position 0.89·route commands 0.72

Robot

Vizbot·LoCoBot·Clearpath Jackal wheeled, Unitree Go1 quadruped (body text mentions 5 types including a drone); Go1·LoCoBot not in training data

Inference

Inference device & speed

Runs at 4 Hz, PD controller tracks waypoints (paper); README deployment example LoCoBot + Jetson Orin Nano; latency not disclosed

Inference memory

Not disclosed

Weights bf16 (calc.)

0.1 GB

Where it runs

Onboard Jetson Orin Nano (LoCoBot, README)

Reinforcement learning & base

Reinforcement learning

Not used · Maximum-likelihood imitation learning only (RL appears only in baselines)

Base control

Wheeled·Quadruped (Waypoints → tracking controller) · Policy outputs 5 waypoints → PD controller tracks them at 4 Hz to generate velocity commands; the robot only needs a low-level velocity-tracking controller

Notes

Key features

Trains a 31M Transformer policy on multi-robot driving data that extends the GNM data mix, and adapts by replacing the goal token with soft prompts for other goal modalities such as GPS coordinates and route commands. Combining diffusion subgoal proposals with graph search, it performed kilometer-scale exploration and also worked zero-shot on a Go1 quadruped and LoCoBot unseen in training.

Notes

Table 6 training resources are a 'representative configuration'; a footnote states that multiple GPU configurations were used. Data scale stated inconsistently (abstract 'hundreds of hours', text 100+ hours, Table 7 caption 150+ hours, total 160h/80h used). The public vint.yaml uses 100 epochs and batch 256, unlike the paper. Weights released on Google Drive. CoRL 2023.

Verification

Confirmed