About two minutes · no cost · no commitment
CorpusMarket Robotics
What is a world model?
A world model is an AI model that predicts what will happen next. Show it the current scene and a possible action, and it produces the scene that would follow. Robotics uses world models as learned simulators, and they are built almost entirely from video.
By Linus Näslund, Co-Founder
- No cost to apply
- Nothing shared before we talk
- Reply within 2 business days
Contents06
The idea in plain words
You already carry a world model in your head. Before you slide a full glass across a table you know roughly what will happen, and you slow down. You did not calculate anything. You have seen enough glasses and tables to predict the result.
In AI, a world model is a model trained to make that kind of prediction. Its input is the current situation, usually camera frames, plus an action. Its output is what comes next. Run it repeatedly and it plays out a whole imagined sequence.
So the short answer to "what is a world model" is a learned simulator. A traditional simulator is built by engineers who write down the laws of physics. A world model learns how things behave by watching.
The idea is not new. A 2018 paper titled "World Models," by David Ha and Jürgen Schmidhuber, showed an agent that was trained entirely inside the imagined environment produced by its own world model, and then carried what it had learned back into the actual environment.
Why world model robotics research matters
Robots are slow and expensive to practice with, and failed attempts break things. Anything that moves practice into software is valuable. World models are used in three ways.
Planning. Before acting, the robot imagines the result of several possible actions and picks the one with the best predicted outcome. Meta describes its V-JEPA 2 model in these terms, as an internal simulator that predicts the outcome of a hypothetical action, and has used it to plan robot movements in settings the robot had not been trained in.
Generating synthetic experience. A world model can produce endless variations of a scene, with different lighting, weather, layouts and objects. These are used as extra training material for the models that control robots, which are covered in our guide to VLA models. NVIDIA describes this as a main use of its Cosmos models, for robots and for self-driving vehicles.
Evaluating policies without a robot. A policy is the model that controls a robot. Testing a new one properly means running it thousands of times, which is impractical on real machines. The robot maker 1X has described training a world model on recordings from its own robots so that it can serve as a virtual simulator for evaluating its policies.
Video is the training material
A world model learns by watching footage and trying to predict what comes next. When its prediction is wrong it adjusts. After enough footage it has absorbed a great deal: objects fall, liquids pour, doors swing on hinges, cloth drapes.
The amounts involved are large. Meta says V-JEPA 2 was first trained on more than a million hours of video, and then on a much smaller amount of robot data (it cites 62 hours) to connect what it had learned to actions.
That two-stage pattern is common. General video teaches how the world behaves. A smaller set of recordings that include actions teaches what the robot's own movements do.
The kind of video matters. Films and online clips are edited, shot from a distance and full of cuts. For a robot that works with its hands, the most useful footage is continuous, close-up and filmed from the worker's own viewpoint, showing hands, tools and materials as a task goes from start to finish. That is the footage described in our guide to egocentric data collection.
World models you may hear about
| Name | From | What its makers say it is |
|---|---|---|
| Cosmos | NVIDIA | An open platform of world foundation models for physical AI, used to generate training data and to simulate for robots and autonomous vehicles |
| Genie 3 | Google DeepMind | A model that generates interactive environments from a text prompt, which can be moved through in real time |
| V-JEPA 2 | Meta | A world model trained on video, released openly, and used to plan robot actions |
| 1X World Model | 1X | A world model trained on the company's own robot recordings, used to evaluate its robot policies |
"NVIDIA Cosmos" and "world foundation model" often appear together. A world foundation model is simply a large world model meant as a base that others adapt, in the same way a large language model is a base for many applications.
The limits
World models are impressive and unreliable in specific ways.
Physics errors. The model has learned what footage looks like, not the laws behind it. Objects can pass through each other, appear, vanish or change shape. The finer the contact (a screw thread, a zipper, a knot), the more likely the prediction is wrong. These are exactly the details hands-on work depends on.
Compounding drift. Each predicted frame is built on the one before it. Small errors add up, and after a while the imagined scene no longer matches anything real. Google DeepMind says of Genie 3 that its environments stay consistent for a few minutes, not hours, and lists a limited range of actions among its current limits.
Looking right is not being right. A generated clip can be convincing to the eye and still be useless for training a robot if the forces and contacts in it could not happen.
Narrow coverage. A world model predicts well for situations like the ones it watched. Where it has seen little footage, it guesses.
For these reasons world models add to real data and traditional simulation, and do not replace them. How robots learn explains how the pieces fit together.
Real-world video is the raw material
Every limit above comes back to what the model has watched. A world model that has never seen drywall being hung cannot imagine it convincingly, and a robot trained on that imagination inherits the gaps.
This makes real footage of real work valuable twice over. It teaches robots directly, and it teaches the world models that generate further practice for them. It is also why interest in video data licensing has grown, and why labs pay for new recordings where no suitable footage exists. Our page on robot training data covers the full range of sources.
Most hands-on work has never been filmed from the worker's point of view. If your business does that kind of work, tell us about your business. CorpusMarket checks the fit and can introduce you to a recording programme run by our partner Amelia Robotics.
Questions and answers
Frequently asked questions
What is a world model in AI?
A model that has learned how a scene changes over time and in response to actions. Given what is happening now and what an agent does, it predicts what comes next, usually as video or as a compact internal description.
What is a world foundation model?
A large, general-purpose world model trained on very large amounts of video, meant to be adapted to many uses. NVIDIA uses the term for its Cosmos models, which are aimed at robots and autonomous vehicles.
How is a world model different from a video generator?
A video generator makes footage that looks right. A world model has to respond to actions, so that pushing a cup to the left produces a different future than pushing it to the right. Many world models are built on video generation and add that control.
Why do robots need world models?
To try things without doing them. A robot or its developers can use a world model to compare possible actions, to produce practice scenarios, and to test a new control model without risking a real machine.
More pages
Read next
What is a VLA model?
A VLA model is the kind of AI model that drives most new general-purpose robots. It looks through the robot's cameras, reads an instruction in ordinary language, and outputs the movements to carry it out. How well it does that depends heavily on what it was shown during training.
Read moreHow robots learn: imitation learning and diffusion policy explained
Most robots that do something new today learned it by imitation learning, which means copying recorded examples of the task. This guide explains that method and the others around it without the math, and ends with what each one needs in data.
Read moreVideo data licensing for AI training
AI developers license video from businesses the same way they license text and images. Whether your video is worth licensing depends on what it shows, who is in it and whether you hold the rights. This page separates the two routes, licensing footage you already have and recording new footage that a lab has asked for.
Read moreNext step
Tell us about your business
If your crew does hands-on work, a robotics lab may pay for video of it. Send your details and we will tell you whether it fits and what happens next.
- No cost to apply
- Nothing shared before we talk
- Reply within 2 business days