VLA Gallery / All models / Navigation

CityWalker

New York University · 2024-11

VANavigation214Mregression

Imitation-learns a GPS-goal-conditioned urban navigation policy from thousands of hours of YouTube city walking·driving videos labeled with trajectories via DPVO visual odometry.

Paper

CityWalker: Learning Embodied Urban Navigation from Web-Scale VideosarXiv 2411.17820

Overview

Category

Navigation

Type

VA

Released

2024-11

Organization

New York University

Open weights

Yes

Parameters

214M (127M trainable, DINOv2 frozen)

Architecture

Backbone LLM

None

Vision encoder

DINOv2 ViT-B/14 (frozen)

Action head

5-waypoint regression + arrival prediction head (regression)

Chunk

5 waypoints

Control rate

Not disclosed (RTX 3060 laptop 1.7 fps)

Architecture (original summary)

Frozen DINOv2 ViT-B/14 encodes 5 past frames + Fourier feature embedding of goal coordinates → Transformer 16 layers·8 heads (768-dim) → action head: 5 Euclidean waypoints + arrival prediction head; modified ViNT PD controller issues velocity commands

Training

Pretraining accelerators

2 H100

Pretraining duration

30 hours

Pretraining steps & batch

10 epochs · batch 32

GPU/TPU-hours

60 (Converted · 2× H100 × 30 hours)

Fine-tuning

Fine-tuned on 6 hours of Go1 teleoperation data, lr 5e-5; GPU·time not disclosed

Data type

Web video / real-robot teleoperation

Data scale

2,000 of 2,522 total hours of YouTube walking·driving video used for training; Go1 teleoperation 15 hours (fine-tuning 6h·evaluation 9h)

Main datasets

CityWalk YouTube walking · driving playlists (DPVO labels); in-house Go1 teleoperation data (HF ai4ce/CityWalker)

Tasks & evaluation

Offline urban benchmark average (fine-tuned): MAOE 15.2°·arrival rate 81.8% (ViNT 16.5°·70.5%, GNM 16.2°·68.6%, NoMaD 19.1°·68.6%); real-world Go1 success rate 77.3% (fine-tuned ViNT 57.1%, zero-shot NoMaD 42.9%, zero-shot ViNT 37.7%)

Robot

Unitree Go1 quadruped (Livox Mid-360 LiDAR·webcam)

Inference

Inference device & speed

DINOv2 backbone 1.7 fps on an RTX 3060 laptop GPU (20 fps when replaced with EfficientNet-B0); deployment compute device not stated

Inference memory

Not disclosed

Weights bf16 (calc.)

0.4 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning only (web video + DPVO trajectory labels)

Base control

Quadruped (Waypoints → tracking controller) · Policy outputs 5 waypoints → modified ViNT PD controller converts them into Go1 velocity commands

Notes

Key features

Imitation-learns a GPS-goal-conditioned urban navigation policy from thousands of hours of YouTube city walking·driving videos labeled with trajectories via DPVO visual odometry. Trained on 1,000+ h, it outperformed fine-tuned ViNT zero-shot, and after 6 h of teleoperation fine-tuning reached a 77.3% real-world success rate on Go1 (fine-tuned ViNT 57.1%).

Notes

Share and number of driving videos not stated. Deployment compute device not stated (speed measured on an RTX 3060 laptop). Pretrained models released on GitHub releases. CVPR 2025.

Verification

Confirmed