VLA Gallery / All models / Navigation

LeLaN

UC Berkeley·Toyota Motor North America · 2024-10

VLANavigationNot disclosedregression

Attaches object instructions to YouTube tour·human walking videos and robot driving data with SAM·VLM (CogVLM)·GPT-3.5, labels object positions with depth estimation, and trains a language-conditioned object navigation policy.

Paper

LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild VideosarXiv 2410.03603

Overview

Category

Navigation

Type

VLA

Released

2024-10

Organization

UC Berkeley·Toyota Motor North America

Open weights

Yes

Parameters

Not disclosed

Architecture

Backbone LLM

None

Vision encoder

ResNet-FiLM (prompt-conditioned; history variant EfficientNet-B0)

Action head

Linear·angular velocity 24-step regression (regression)

Chunk

24 steps (first step executed)

Control rate

Not disclosed (0.054 s per action)

Architecture (original summary)

Frozen CLIP text encoder (ViT-B/32) + prompt-FiLM-conditioned ResNet visual encoder (EfficientNet-B0 for the history variant) → Transformer + MLP → regresses a 24-step sequence of linear·angular velocities, executes only the first step (receding horizon)

Training

Pretraining accelerators

Single RTX 4090 (i9·96GB RAM workstation)

Pretraining duration

Not disclosed

Pretraining steps & batch

Batch 256 (epochs·steps Not disclosed; public lelan.yaml 100 epochs)

GPU/TPU-hours

Not disclosed

Fine-tuning

Collision-avoidance variant: pretrained without collision loss, then fine-tuned with a NoMaD-supervised collision loss (README); GPU and time not disclosed

Data type

Real-robot teleoperation / real-robot autonomous driving logs / web video / human video

Data scale

Table 1: indoor robot 31.0h (111,570 images) · YouTube tours 82.5h (593,727) · human walking 15.7h (113,277); prompts 3.4M, 14.6M, 2.3M

Main datasets

GO Stanford 2 · 4; SACSoN (HuRoN); YouTube Tour (32 countries); Human-walking (in-house collection, 11 cities in 3 countries)

Tasks & evaluation

Real-world success rate reaching language-specified objects 0.89 (OWL-v2+ZoeDepth 0.67, CoW 0.58, OWL-ViT+ViNT 0.47, OpenFMNav 0.43); noisy prompts 0.88; dynamic objects 0.83; obstacle environment with collision loss 0.60 (OWL-v2+ZoeDepth 0.40)

Robot

Wheeled robot (README: Roomba-based Vizbot; RICOH Theta S front fisheye), Unitree Go1 quadruped; cross-camera tests with D435i, etc.

Inference

Inference device & speed

Action generation per observation 0.054 s onboard Jetson Orin (AGX); first of 24 steps executed

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Onboard Jetson Orin AGX

Reinforcement learning & base

Reinforcement learning

Not used · Supervised learning only (distills NoMaD collision-avoidance trajectories through the loss)

Base control

Wheeled·Quadruped (Direct base velocity/pose output) · Policy outputs a 24-step sequence of linear and angular velocities and executes only the first step via /cmd_vel (receding horizon)

Notes

Key features

Attaches object instructions to YouTube tour·human walking videos and robot driving data with SAM·VLM (CogVLM)·GPT-3.5, labels object positions with depth estimation, and trains a language-conditioned object navigation policy. Real-world zero-shot success rate 0.89 beat modular baselines such as OWL-v2+ZoeDepth (0.67)·CoW (0.58), and it runs at 0.054 s per action on Jetson Orin.

Notes

Total data: 130+ hours in abstract (120+ hours in contribution list; three entries in Table 1 sum to 129.2h). Parameters·training time Not disclosed. The arXiv badge in the README points to a different paper (2407.08693); the actual paper is 2410.03603. Long-range goals combine OWL-ViT2 node search + ViNT topological graph (README). Data·weights released. CoRL 2024.

Verification

Corrections applied