VLA Gallery / All models / VA

HPT

MIT CSAIL; Meta FAIR · 2024-09

VAVATrunk Small 3.1M, Base 12.6M, Large 50.5M, XLarge 226.8M, Huge 1.1Bregression

Pretrains on heterogeneous robot·sim·human video·deployed robot data with a shared trunk to share representations across embodiments.

Paper

Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained TransformersarXiv 2409.20537

Overview

Category

VA

Type

VA

Released

2024-09

Organization

MIT CSAIL; Meta FAIR

Open weights

Partial

Parameters

Trunk Small 3.1M, Base 12.6M, Large 50.5M, XLarge 226.8M, Huge 1.1B; Base (with language) 50.6M (README)

Architecture

Backbone LLM

None

Vision encoder

ResNet18 stem (frozen)

Action head

MLP regression head (regression)

Chunk

8 steps

Control rate

Not disclosed

Architecture (original summary)

Per-embodiment stem (frozen pretrained ResNet18 features + proprioception → fixed number of tokens, e.g. 16) + shared trunk transformer + head (default MLP regression; transformer decoder·diffusion heads also used); action horizon 8, observation horizon 4

Training

Pretraining accelerators

V100 8–128 GPUs (full experimental range, no breakdown by model size)

Pretraining duration

Half a day–1 month (full experimental range)

Pretraining steps & batch

Base: 80k iterations·batch 256 (about 0.65B tokens); scaled: batch 2048 (iteration count not captured)

GPU/TPU-hours

Not disclosed

Fine-tuning

Real: 1 RTX 2080Ti per task, 20,000 iterations·batch 256, about 4 h (HPT-Small trunk, stem·head re-initialized); sim: 20,000 iterations·batch 512·lr 1e-5 (GPU not stated)

Data type

Real-robot teleoperation; simulation·synthetic; human video; other (deployed robots FrodoBot-2k)

Data scale

Base: 27 datasets, 16k trajectories, 5M samples; scaled: 52 datasets, 270k (Table 2)/200k (main text·project page) trajectories, 155M samples; about 100 demonstrations per fine-tuning task

Main datasets

42 Open-X datasets (including DROID); 7 simulators (Drake, MuJoCo, Isaac Sim, PyBullet, etc.); EPIC-Kitchens, PoCo; FrodoBot-2k

Tasks & evaluation

Sim: Meta-World, RoboMimic, Fleet-Tools, Simpler; real: Sweep Leftover, Fill Water, Scoop Food, Switch Insertion (2 embodiments: pet care·assembly)

Robot

Pretraining on many embodiments (Open-X etc.); real fine-tuning on 2 single-arm types

Inference

Inference device & speed

HPT-Base 47 Hz, HPT-XL 19 Hz on RTX 3070

Inference memory

Not disclosed

Weights bf16 (calc.)

2.2 GB

Where it runs

Local PC

Reinforcement learning & base

Reinforcement learning

Not used · Supervised pretraining + imitation learning fine-tuning

Base control

Fixed · No base control (real fine-tuning on a single arm)

Notes

Key features

Pretrains on heterogeneous robot·sim·human video·deployed robot data with a shared trunk to share representations across embodiments. The only profiled VA policy with a separate pretraining stage.

Notes

Parameters (B) is the Huge trunk 1.1B (excluding stem and head; Huge weights not released). Human video trajectories: App. A.1 says 2,000 vs §4.3 says 300 (different sections). Language is optional (T5 tokens) and off in the base model. Real experiments tried both MLP and diffusion heads. NeurIPS 2024 Spotlight.

Verification

Confirmed