VLA Gallery / All models / Navigation

TrackVLA

Peking University·Galbot·Beihang University·Beijing Normal University·BAAI · 2025-05

VLANavigation7Bdiffusion

An embodied visual tracking VLA that places a language head for target recognition and an anchor-based diffusion head for trajectory planning in parallel on a shared LLM, jointly learning recognition and tracking trajectory planning.

Paper

TrackVLA: Embodied Visual Tracking in the WildarXiv 2505.23189

Overview

Category

Navigation

Type

VLA

Released

2025-05

Organization

Peking University·Galbot·Beihang University·Beijing Normal University·BAAI

Open weights

No

Parameters

7B (based on Vicuna-7B; overall figure not stated)

Architecture

Backbone LLM

Vicuna-7B

Vision encoder

EVA-CLIP (frozen)

Action head

anchor-based diffusion head (DiT, DDIM 2 steps) waypoints + language head (perception) (diffusion)

Chunk

10 waypoints

Control rate

10 FPS

Architecture (original summary)

EVA-CLIP (frozen) + 2-layer MLP projector + Vicuna-7B; 64 tokens for the current frame · 4 tokens per past frame (32-frame window); for tracking, an anchor-based diffusion head (DiT 12 layers · hidden 768 · 12 heads, 40 anchors, DDIM 2 steps) generates 10 waypoints (x, y, θ), language modeling head for recognition; 10 FPS in the real world

Training

Pretraining accelerators

24 H100

Pretraining duration

About 15 h

Pretraining steps & batch

1 epoch · total batch 196 (lr 2e-5 cosine)

GPU/TPU-hours

360 (Stated · H100-hours ("totaling 360 GPU hours"))

Fine-tuning

N/A

Data type

Simulation·synthetic / web vision-language

Data scale

1.7M total: embodied visual tracking 855K (885K in Sec. 4.1) + video QA 855K (person recognition 362K + open-ended VQA 493K)

Main datasets

EVT-Bench (Habitat 3.0, 804 HM3D · MP3D scenes, 100 human avatars); SYNTH-PEDES; public VQA

Tasks & evaluation

EVT-Bench SR single target 85.1% · distracted target 57.6% · ambiguous target 50.2%; Gym-UnrealCV zero-shot SR 1.00 (single) · 0.91 (distractors) · 1.00 (unseen objects); real-world 10 runs per difficulty SR 100% · 90% · 70% (commercial tracking drone DJI Flip 100% · 70% · 50%)

Robot

Unitree Go2 quadruped + RealSense D455 (640×480, HFOV 90°); pure pursuit closed-loop tracking, LiDAR elastic band obstacle avoidance

Inference

Inference device & speed

Remote server RTX 4090, 10 FPS; action model (Ours-Base) 13 ms; memory not disclosed

Inference memory

Not disclosed

Weights bf16 (calc.)

14 GB

Where it runs

Remote server (RTX 4090)

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning (SFT) only

Base control

Quadruped (Waypoints → tracking controller) · VLA outputs waypoint trajectory → pure pursuit closed-loop tracking (with LiDAR-based obstacle avoidance)

Notes

Key features

An embodied visual tracking VLA that places a language head for target recognition and an anchor-based diffusion head for trajectory planning in parallel on a shared LLM, jointly learning recognition and tracking trajectory planning. Trained on its own EVT-Bench sim tracking data and 1.7M recognition samples, it showed robust tracking at 10 FPS in zero-shot Gym-UnrealCV and the real world.

Notes

Tracking sample count inconsistent in the text (855K vs 885K); TrackVLA weights not released (only EVT-Bench and Uni-NaVid evaluation code and weights released); EVA-CLIP and Vicuna-7B initialized from pretrained weights; CoRL 2025.

Verification

Confirmed