About two minutes · no cost · no commitment
CorpusMarket Robotics
What is a VLA model?
A VLA model is the kind of AI model that drives most new general-purpose robots. It looks through the robot's cameras, reads an instruction in ordinary language, and outputs the movements to carry it out. How well it does that depends heavily on what it was shown during training.
By Linus Näslund, Co-Founder
- No cost to apply
- Nothing shared before we talk
- Reply within 2 business days
Contents06
The short definition
A VLA model, short for vision-language-action model, is one neural network with three jobs. It sees (vision), it understands an instruction (language), and it moves a robot (action).
Give it a camera image of a messy counter and the words "put the cups in the sink." It outputs a short burst of motor commands. The robot moves, the camera shows the new scene, and the model produces the next burst. This repeats several times a second until the task is done.
Before VLA models, these were separate systems built by separate teams: one to recognize objects, one to plan, one to control the arm. A VLA model learns all of it together from examples.
How a vision-language-action model extends a vision-language model
A vision-language model, or VLM, is the kind of AI that can look at a photo and answer questions about it. It learned from a very large number of images and captions on the internet. It knows what a cup is, what a sink is, and what "in" means.
What it cannot do is move anything. Its only output is text.
A VLA model fixes that by teaching the same model a new kind of output. Google DeepMind's RT-2, which its makers introduced as a vision-language-action model, did this in a simple way. It wrote robot actions out as short strings of text and had the model produce them the way it would produce words. The model was then trained on web data and robot data together, so it kept what it knew about the world while learning to act.
Later models changed the details. Some add a separate part that generates smooth motion with diffusion or a related technique (see how robots learn for what that means). The principle is the same: start from a model that understands images and language, then train it on recordings of physical tasks.
What VLA models are trained on
There are two layers of training data.
Internet images and text. This gives the model general knowledge. It is plentiful and was collected long before anyone thought about robots.
Recordings of physical tasks. This is where the model learns to act, and it comes in three forms:
- Robot demonstrations. A person steers a robot through a task while everything is recorded. The largest shared collection, Open X-Embodiment, pools more than a million robot recordings from 22 kinds of robot.
- Video of people. First-person footage of hands doing real tasks. It does not contain robot motor commands, but it shows how work is done across far more places than robots have ever been.
- Synthetic data. Examples generated in simulation or by world models.
The first layer is huge and the second is small. Our guide to robot training data explains why the second layer is so hard to grow.
Well-known VLA models
| Model | Released by | Availability | In one line |
|---|---|---|---|
| RT-2 | Google DeepMind | Research model | Trained on web data and robot data together, with actions written out like words |
| OpenVLA | Researchers at Stanford, UC Berkeley, Toyota Research Institute, Google DeepMind, Physical Intelligence and MIT | Open source | A 7-billion-parameter model trained on about 970,000 robot recordings from Open X-Embodiment |
| π0 (pi0) | Physical Intelligence | Open code and weights | Built on a pretrained vision-language model, generates motion with a technique called flow matching |
| GR00T | NVIDIA | Open models | Models for humanoid robots, trained on human video, teleoperated robot data and synthetic data |
| Gemini Robotics | Google DeepMind | Limited access for testers | Google DeepMind's current vision-language-action models, including a version that runs on the robot itself |
| Helix | Figure | Used on Figure's own robots | Controls the upper body of Figure's humanoid, fingers included, and runs on the robot's onboard computers |
| SmolVLA | Hugging Face | Open source | A small model trained on datasets shared by the robotics community, made to run on consumer hardware |
This list changes quickly. New versions appear every few months, and the names above are families more than single models.
What VLA models can and cannot do today
It is worth being careful here, because demonstration videos show the best runs.
What they do reasonably well. They follow simple spoken or written instructions. They handle objects they were never specifically trained on, because the vision-language layer recognizes them. They perform short tabletop tasks such as picking, placing, sorting, wiping and opening drawers. With extra training on one task they can manage longer and fiddlier jobs. Physical Intelligence, for example, has shown laundry folding and box assembly.
What they do poorly. They are slow next to a person. They fail more often than any employer would accept from a worker, and they fail in unfamiliar ways. They struggle with long jobs made of many steps, with tight spaces, with tools, and with anything that needs force and feel, such as starting a screw or seating a pipe fitting. They do not know when they are about to make a mistake.
What usually gets left out. A model that works on one robot in one lab often does not work on another robot somewhere else without new data. Physical Intelligence says as much in the notes published with its open models: they may or may not work on your particular robot.
The gap between a tidy lab bench and a real workplace is the central problem. A VLA model has seen few kitchens during a dinner rush and few crawl spaces.
Why VLA models are hungry for varied real-world data
A language model becomes general by reading text about everything. A VLA model becomes general by seeing physical tasks done everywhere, and that record does not exist yet.
Three things matter more than raw volume:
- Variety of places. Many different kitchens teach more than one kitchen filmed many times.
- Variety of people. Different hands, heights and habits.
- Real tasks at real speed. Staged work in a studio leaves out the interruptions and fixes that make up a working day.
Robots steered by hand cannot supply this, because there are too few robots and they are not in real workplaces. So labs are turning to first-person video recorded by people on the job. The public egocentric video datasets were a start, and paid recording programmes are the next step.
Labs that need this kind of footage can read our page for robotics labs. If you run a business where people work with their hands, that work is what these models have seen least. Tell us about your business and we will check whether it fits the recording programme run by our partner Amelia Robotics.
Questions and answers
Frequently asked questions
What is a VLA model?
VLA stands for vision-language-action. It is a single AI model that takes camera images and a written or spoken instruction as input and produces robot movements as output.
How is a vision-language-action model different from a chatbot?
A chatbot answers in text. A vision-language-action model answers in motion, as a stream of commands to a robot's joints and gripper. It is usually built on the same kind of model as a chatbot and then trained further on recordings of physical tasks.
Is OpenVLA free to use?
OpenVLA is published as open source by its authors, with the model and training code available to download. Getting any such model to work well on a particular robot still takes that robot's own data and engineering time.
What are pi0 and RT-2?
RT-2 is a research model from Google DeepMind, introduced as a vision-language-action model that learns from both web data and robot data. Pi0, written π0, is a model from Physical Intelligence that the company has published with open code and weights.
Can a VLA model learn from video of people?
Yes, and several are trained that way in part. Video of people shows how tasks are done across many settings, and robot recordings then tie that knowledge to a particular robot body. NVIDIA lists human video among the training sources for its GR00T models.
More pages
Read next
How robots learn: imitation learning and diffusion policy explained
Most robots that do something new today learned it by imitation learning, which means copying recorded examples of the task. This guide explains that method and the others around it without the math, and ends with what each one needs in data.
Read moreWhat is a world model?
A world model is an AI model that predicts what will happen next. Show it the current scene and a possible action, and it produces the scene that would follow. Robotics uses world models as learned simulators, and they are built almost entirely from video.
Read moreEgocentric video datasets
A small number of public datasets come up in almost every paper on robots learning from video. Some are first-person recordings of people. Others are recordings made on robots. This page lists the well-known ones, says what each contains, and explains why labs still pay for new footage.
Read moreNext step
Tell us about your business
If your crew does hands-on work, a robotics lab may pay for video of it. Send your details and we will tell you whether it fits and what happens next.
- No cost to apply
- Nothing shared before we talk
- Reply within 2 business days