OpenVLA

Table of contents

  1. ▪ Architecture
  2. ▪ Training Data
  3. ▪ Fine-Tuning and Inference

OpenVLA is a 7B-parameter open-source vision-language-action model for generalist robotic manipulation. It follows the RT-2 recipe — fine-tuning a web-pretrained VLM to autoregressively generate discretized robot actions — but is fully open: model weights, training code, and data mixtures are all publicly available.

▪ Architecture

OpenVLA is built on top of the Prismatic VLM:

  • Vision backbone: a fused DINOv2 + SigLIP encoder at 224px resolution, combining low-level spatial features with high-level semantic features.
  • Language model: a Llama-2 7B backbone.
  • Action representation: each dimension of the 7-DoF end-effector action is discretized into 256 bins and mapped to tokens in the Llama tokenizer, so actions are generated by standard next-token prediction.

▪ Training Data

The flagship openvla-7b model was trained on the Open-X Magic Soup++ mixture: about 970k real robot trajectories from the Open X-Embodiment dataset, filtered for single-arm manipulation with language annotations.

▪ Fine-Tuning and Inference

  • LoRA fine-tuning: adapts the model to a new robot setup with ~100 demonstrations on a single GPU (~27 GB VRAM).
  • Full fine-tuning: all 7.5B parameters via PyTorch FSDP on a multi-GPU node.
  • Inference: a lightweight Hugging Face transformers interface (AutoModelForVision2Seq) loads the checkpoint and predicts actions with vla.predict_action(...).

Two notable follow-up recipes improve on the base model:

  • FAST: a compression-based action tokenizer that encodes action chunks into few discrete tokens, speeding up autoregressive inference by up to 15x.
  • OFT (Optimized Fine-Tuning): a fine-tuning recipe using continuous actions and parallel decoding, giving 25–50x faster inference, higher success rates, and support for multiple input images and high-frequency bimanual control.

Reference:

This site uses Just the Docs, a documentation theme for Jekyll.