VLA Gallery / All models / Autoregressive & latent-action VLA

TraceVLA

University of Maryland·Microsoft Research · 2024-12

VLAAutoregressive & latent-action VLA7Bdiscrete tokens

Improves spatiotemporal awareness with visual trace prompting, feeding point-tracking trajectories as image prompts.

Paper

TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic PoliciesarXiv 2412.10345

Overview

Category

Autoregressive & latent-action VLA

Type

VLA

Released

2024-12

Organization

University of Maryland·Microsoft Research

Open weights

Yes

Parameters

7B (based on OpenVLA); 4B Phi-3V variant

Architecture

Backbone LLM

Llama 2 7B (OpenVLA) / Phi-3-Vision 4B

Vision encoder

SigLIP + DINOv2 + CoTracker trajectory image

Action head

256-bin tokens (discrete tokens)

Chunk

Not disclosed

Control rate

Not disclosed

Architecture (original summary)

OpenVLA (Prismatic-7B) or Phi-3-Vision 4B; extra image input overlaid with CoTracker visual traces, OpenVLA-style discrete action tokens

Training

Pretraining accelerators

N/A (7B starts from the OpenVLA checkpoint, 4B from Phi-3-Vision weights)

Pretraining duration

N/A

Pretraining steps & batch

N/A

GPU/TPU-hours

Not disclosed

Fine-tuning

7B: 64 GPUs (README script, 8 nodes × 8), 5 more epochs from OpenVLA; Phi3-4B: 32 H100, 30 epochs, batch 4096; time not disclosed

Data type

Real-robot teleoperation (including visual trajectory annotations)

Data scale

About 150K trajectories + 120 WidowX-250 demonstrations

Main datasets

BridgeData V2, Fractal

Tasks & evaluation

SimplerEnv 137 settings 47.7% (OpenVLA 40.2%); real WidowX-250 4 tasks each seen·unseen

Robot

WidowX-250, SimplerEnv simulation

Inference

Inference device & speed

OpenVLA speed + about 0.002 s for extra tokens, CoTracker (M=5) 0.03 s per step

Inference memory

Not disclosed

Weights bf16 (calc.)

14 GB

Where it runs

Not disclosed

Reinforcement learning & base

Reinforcement learning

Not used · Imitation learning (behavior cloning) only

Base control

Fixed · No base control (WidowX arm)

Notes

Key features

Improves spatiotemporal awareness with visual trace prompting, feeding point-tracking trajectories as image prompts. Scored 47.7% across 137 SimplerEnv settings, ahead of OpenVLA (40.2%).

Notes

Continues training OpenVLA on trajectory-annotated data, so there is no separate robot pretraining. Parameters (B) refer to the 7B variant. One extraction suggested Capital One is also among the v3 PDF affiliations†.

Verification

Confirmed