VLA Gallery / All models / Navigation

Mobility VLA

Google DeepMind · 2024-07

VLANavigationNot disclosedOther

Training-free hierarchical navigation policy that feeds an entire environment demonstration tour video into a long-context VLM (Gemini 1.5 Pro) to find the goal frame matching text/image instructions, and plans low-level waypoints on a topological graph built from the same tour.

Paper

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological GraphsarXiv 2407.07775

Overview

Category

Navigation

Type

VLA

Released

2024-07

Organization

Google DeepMind

Open weights

No

Parameters

Not disclosed (Gemini 1.5 Pro)

Architecture

Backbone LLM

Gemini 1.5 Pro (cloud API)

Vision encoder

Built into Gemini 1.5 Pro (details not disclosed)

Action head

Target frame index → topological-graph waypoints (Δx · Δy · Δθ) (Other)

Chunk

None (1-step waypoint)

Control rate

High level 10–30 s per query · low level 0.19 s/step (sim)

Architecture (original summary)

High level: Gemini 1.5 Pro long-context VLM takes all 948 frames (1 Hz) of the tour video + multimodal instruction and outputs a goal frame index (10–30 s per query, cloud) / low level: shortest path on COLMAP-based topological graph → robot-centric waypoint actions (Δx, Δy, Δθ) → MPC tracking; low-level policy 0.19 s/step (sim)

Training

Pretraining accelerators

N/A (no training, uses Gemini 1.5 Pro API)

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

N/A

Data type

Real-robot teleoperation

Data scale

No training data; uses 948 frames of gamepad-teleoperated tour video (about 16 minutes, 1 Hz) as environment context

Main datasets

No training dataset; evaluated on in-house MINT instructions (incl. 57 crowdsourced)

Tasks & evaluation

Real office goal search SR: no reasoning needed 95%·reasoning needed 86%·small objects 42%·multimodal 90% (Table 3); end-to-end SR 80%·80%·40%·85%, SPL 0.59·0.69·0.38·0.64 (Table 2, 5 instructions per category×4 start points); sim end-to-end SR 90%·SPL 0.84; tested in a home-like environment (smartphone tour)

Robot

Wheeled mobile manipulator (model Not stated), RGB cameras only

Inference

Inference device & speed

VLM runs in the cloud (uploads 948 tour images per call), 10–30 s per goal-index search query; low-level policy 0.19±0.047 s/step (sim); low onboard compute requirement

Inference memory

Not disclosed

Weights bf16 (calc.)

Not disclosed

Where it runs

Cloud (Gemini 1.5 Pro) + robot MPC

Reinforcement learning & base

Reinforcement learning

Not used · No training (Gemini 1.5 Pro zero-shot + topological graph planning)

Base control

Wheeled (Waypoints → tracking controller) · VLM selects goal frame → waypoints (Δx·Δy·Δθ) along topological graph path tracked with MPC

Notes

Key features

Training-free hierarchical navigation policy that feeds an entire environment demonstration tour video into a long-context VLM (Gemini 1.5 Pro) to find the goal frame matching text/image instructions, and plans low-level waypoints on a topological graph built from the same tour. In an 836 m² real office it reports goal-finding success of 86% and 90% for reasoning-required and multimodal instructions, 26% and 60% higher than prior methods.

Notes

86% · 90% are goal-finding rates (Table 3); end-to-end success (Table 2) is 80% for reasoning-required · 85% multimodal, differing from the narrative; architecture per the MoE Transformer description in the Gemini 1.5 report; Gemini parameters and inference hardware not disclosed; code and weights not released.

Verification

Confirmed