CorpusMarket Robotics

What is a VLA model?

A VLA model is the kind of AI model that drives most new general-purpose robots. It looks through the robot's cameras, reads an instruction in ordinary language, and outputs the movements to carry it out. How well it does that depends heavily on what it was shown during training.

By Linus Näslund, Co-Founder

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days
Contents06
  1. 01The short definition
  2. 02How a vision-language-action model extends a vision-language model
  3. 03What VLA models are trained on
  4. 04Well-known VLA models
  5. 05What VLA models can and cannot do today
  6. 06Why VLA models are hungry for varied real-world data

The short definition

A VLA model, short for vision-language-action model, is one neural network with three jobs. It sees (vision), it understands an instruction (language), and it moves a robot (action).

Give it a camera image of a messy counter and the words "put the cups in the sink." It outputs a short burst of motor commands. The robot moves, the camera shows the new scene, and the model produces the next burst. This repeats several times a second until the task is done.

Before VLA models, these were separate systems built by separate teams: one to recognize objects, one to plan, one to control the arm. A VLA model learns all of it together from examples.

How a vision-language-action model extends a vision-language model

A vision-language model, or VLM, is the kind of AI that can look at a photo and answer questions about it. It learned from a very large number of images and captions on the internet. It knows what a cup is, what a sink is, and what "in" means.

What it cannot do is move anything. Its only output is text.

A VLA model fixes that by teaching the same model a new kind of output. Google DeepMind's RT-2, which its makers introduced as a vision-language-action model, did this in a simple way. It wrote robot actions out as short strings of text and had the model produce them the way it would produce words. The model was then trained on web data and robot data together, so it kept what it knew about the world while learning to act.

Later models changed the details. Some add a separate part that generates smooth motion with diffusion or a related technique (see how robots learn for what that means). The principle is the same: start from a model that understands images and language, then train it on recordings of physical tasks.

What VLA models are trained on

There are two layers of training data.

Internet images and text. This gives the model general knowledge. It is plentiful and was collected long before anyone thought about robots.

Recordings of physical tasks. This is where the model learns to act, and it comes in three forms:

  • Robot demonstrations. A person steers a robot through a task while everything is recorded. The largest shared collection, Open X-Embodiment, pools more than a million robot recordings from 22 kinds of robot.
  • Video of people. First-person footage of hands doing real tasks. It does not contain robot motor commands, but it shows how work is done across far more places than robots have ever been.
  • Synthetic data. Examples generated in simulation or by world models.

The first layer is huge and the second is small. Our guide to robot training data explains why the second layer is so hard to grow.

Well-known VLA models

ModelReleased byAvailabilityIn one line
RT-2Google DeepMindResearch modelTrained on web data and robot data together, with actions written out like words
OpenVLAResearchers at Stanford, UC Berkeley, Toyota Research Institute, Google DeepMind, Physical Intelligence and MITOpen sourceA 7-billion-parameter model trained on about 970,000 robot recordings from Open X-Embodiment
π0 (pi0)Physical IntelligenceOpen code and weightsBuilt on a pretrained vision-language model, generates motion with a technique called flow matching
GR00TNVIDIAOpen modelsModels for humanoid robots, trained on human video, teleoperated robot data and synthetic data
Gemini RoboticsGoogle DeepMindLimited access for testersGoogle DeepMind's current vision-language-action models, including a version that runs on the robot itself
HelixFigureUsed on Figure's own robotsControls the upper body of Figure's humanoid, fingers included, and runs on the robot's onboard computers
SmolVLAHugging FaceOpen sourceA small model trained on datasets shared by the robotics community, made to run on consumer hardware

This list changes quickly. New versions appear every few months, and the names above are families more than single models.

What VLA models can and cannot do today

It is worth being careful here, because demonstration videos show the best runs.

What they do reasonably well. They follow simple spoken or written instructions. They handle objects they were never specifically trained on, because the vision-language layer recognizes them. They perform short tabletop tasks such as picking, placing, sorting, wiping and opening drawers. With extra training on one task they can manage longer and fiddlier jobs. Physical Intelligence, for example, has shown laundry folding and box assembly.

What they do poorly. They are slow next to a person. They fail more often than any employer would accept from a worker, and they fail in unfamiliar ways. They struggle with long jobs made of many steps, with tight spaces, with tools, and with anything that needs force and feel, such as starting a screw or seating a pipe fitting. They do not know when they are about to make a mistake.

What usually gets left out. A model that works on one robot in one lab often does not work on another robot somewhere else without new data. Physical Intelligence says as much in the notes published with its open models: they may or may not work on your particular robot.

The gap between a tidy lab bench and a real workplace is the central problem. A VLA model has seen few kitchens during a dinner rush and few crawl spaces.

Why VLA models are hungry for varied real-world data

A language model becomes general by reading text about everything. A VLA model becomes general by seeing physical tasks done everywhere, and that record does not exist yet.

Three things matter more than raw volume:

  1. Variety of places. Many different kitchens teach more than one kitchen filmed many times.
  2. Variety of people. Different hands, heights and habits.
  3. Real tasks at real speed. Staged work in a studio leaves out the interruptions and fixes that make up a working day.

Robots steered by hand cannot supply this, because there are too few robots and they are not in real workplaces. So labs are turning to first-person video recorded by people on the job. The public egocentric video datasets were a start, and paid recording programmes are the next step.

Labs that need this kind of footage can read our page for robotics labs. If you run a business where people work with their hands, that work is what these models have seen least. Tell us about your business and we will check whether it fits the recording programme run by our partner Amelia Robotics.

Questions and answers

Frequently asked questions

What is a VLA model?

VLA stands for vision-language-action. It is a single AI model that takes camera images and a written or spoken instruction as input and produces robot movements as output.

How is a vision-language-action model different from a chatbot?

A chatbot answers in text. A vision-language-action model answers in motion, as a stream of commands to a robot's joints and gripper. It is usually built on the same kind of model as a chatbot and then trained further on recordings of physical tasks.

Is OpenVLA free to use?

OpenVLA is published as open source by its authors, with the model and training code available to download. Getting any such model to work well on a particular robot still takes that robot's own data and engineering time.

What are pi0 and RT-2?

RT-2 is a research model from Google DeepMind, introduced as a vision-language-action model that learns from both web data and robot data. Pi0, written π0, is a model from Physical Intelligence that the company has published with open code and weights.

Can a VLA model learn from video of people?

Yes, and several are trained that way in part. Video of people shows how tasks are done across many settings, and robot recordings then tie that knowledge to a particular robot body. NVIDIA lists human video among the training sources for its GR00T models.

Next step

Tell us about your business

If your crew does hands-on work, a robotics lab may pay for video of it. Send your details and we will tell you whether it fits and what happens next.

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days

CorpusMarket is independent of Amelia Robotics and may receive a referral fee if your business joins their programme. Nothing on this page is a promise of income.