CorpusMarket Robotics

How robots learn: imitation learning and diffusion policy explained

Most robots that do something new today learned it by imitation learning, which means copying recorded examples of the task. This guide explains that method and the others around it without the math, and ends with what each one needs in data.

By Linus Näslund, Co-Founder

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days
Contents08
  1. 01Programming a robot versus teaching one
  2. 02Imitation learning and behavior cloning
  3. 03Why small errors compound
  4. 04Diffusion policy
  5. 05Reinforcement learning
  6. 06Simulation and the sim-to-real gap
  7. 07How these combine with large pretrained models
  8. 08What each method needs in data

Programming a robot versus teaching one

For most of their history robots were programmed. An engineer wrote out each position the arm should move through, and the robot repeated it exactly. This works when the part arrives in the same spot every time. It fails the moment something is out of place.

Robot learning takes a different route. The robot is shown examples, or allowed to practice, and a model works out the rule that connects what the camera sees to what the motors should do. That model is called a policy. Everything below is a way of producing a good policy.

Imitation learning and behavior cloning

Imitation learning, in the robotics sense, means learning a task from demonstrations. (The same phrase is used for how children copy adults. This page is about robots.)

A demonstration is a recording that pairs what was seen with what was done. The simplest way to learn from it is called behavior cloning. Take thousands of recorded moments. For each one, train the model to output the action the demonstrator took. After enough examples the model produces sensible actions for situations that resemble the recordings.

Behavior cloning is popular because it is simple and it works. There are no rewards to design and no trial and error. If you have good demonstrations, you can train a policy.

Demonstrations come from two main places. A person can steer the robot through the task, which is called teleoperation. Or a person can simply be filmed doing the task with their own hands. The trade-offs are covered in teleoperation versus egocentric video.

Why small errors compound

Behavior cloning has a well-known weakness. The demonstrations show the task going right. They rarely show what to do when it has started to go wrong.

A policy is never perfect, so at some point it places the gripper a little off. Now the camera shows a situation that was not in the recordings. The policy guesses, and the guess is a little worse. Each action changes what the robot sees next, so mistakes feed on themselves. Researchers have described this formally: learning from demonstrations breaks the usual assumption that training examples are independent of one another.

There are three common answers:

  • More varied demonstrations, including recoveries from small slips.
  • Corrections. Let the robot run, have a person step in when it drifts, and add those corrections to the training data.
  • Predicting several steps at once, so the robot commits to a short, coherent motion and does not second-guess itself every instant.

The third answer leads to the next method.

Diffusion policy

Diffusion is the technique behind image generators. Such a model starts with random noise and cleans it up, step by step, until a picture appears. Diffusion Policy, published in 2023 by researchers at Columbia University, Toyota Research Institute and MIT, applies the same idea to robot motion.

The policy looks at the camera image and starts with a random, meaningless sequence of actions. It then refines that sequence over several steps until it becomes a smooth, short plan of movement that fits the scene. The robot carries out part of the plan, looks again, and repeats.

Two things make this useful, and the authors point to both.

First, many tasks have more than one correct way to do them. You can go around an obstacle on the left or the right. A simple policy trained on both kinds of example tends to average them and drive straight into the obstacle. A diffusion policy settles on one valid option.

Second, it predicts a sequence of actions and not a single one. That gives steadier motion.

It is still imitation learning. Diffusion policy is a better way to copy demonstrations, and it needs demonstrations to copy.

Reinforcement learning

Reinforcement learning is learning by trial and error. The robot tries something, receives a score for how well it went, and adjusts to raise the score. Nobody shows it the answer.

Its strength is that it can find solutions no person demonstrated and can exceed the demonstrator. Its weakness in robotics is the amount of practice needed. Millions of attempts are routine, and a real robot is slow, wears out and breaks things when it fails. Someone also has to define the score, which is hard for a task like "tidy the kitchen."

So reinforcement learning in robotics is done mostly in simulation, or used to polish a skill that imitation learning has already taught.

Simulation and the sim-to-real gap

A simulator is a virtual world with physics. Inside it a thousand copies of a robot can practice at once, faster than real time, with no damage. Walking and balance are often learned this way.

The catch is known as the sim-to-real gap, often written sim2real. A simulator is an approximation. Friction, the give of a rubber seal, the way cloth folds and the glare on a steel counter are all a little off. A policy that is flawless in simulation can fail on the real machine.

Engineers narrow the gap by varying the simulation at random (different weights, lighting and textures) so the policy cannot rely on any one detail. This works well for rigid objects and locomotion. It works less well for soft, messy, hand-scale work such as food, fabric and wiring, where the physics is hardest to simulate. Newer world models try to learn the simulator itself from video.

How these combine with large pretrained models

The current approach is to stack the methods.

It starts with a large model that has already learned from internet images and text, so it knows what a mug is and what "put it in the sink" means. That model is then trained to output robot actions, which makes it a vision-language-action model. Our guide to VLA models explains them. Some of these models generate their actions with diffusion or a closely related technique called flow matching, which Physical Intelligence uses in its π0 model.

The training data is mixed on purpose. NVIDIA, for example, says its GR00T models learn from human video, teleoperated robot data and synthetic data from simulation. Reinforcement learning may then tune the result.

What each method needs in data

MethodWhat it needsWhere that comes from
Behavior cloning and diffusion policyDemonstrations of the task, many and variedTeleoperation, and video of people working
Reinforcement learningA way to score attempts, and huge numbers of attemptsSimulation, plus some real practice
SimulationAccurate virtual scenes and objectsEngineering work, and real footage to compare against
Large pretrained modelsBroad knowledge of the world, then examples of physical tasksInternet images and text, then robot data and human video

Three of the four rows depend directly on recordings of real tasks. Simulation and trial and error help, but the knowledge of how a job is done has to come from somewhere, and it comes from people doing it.

That is why robot training data is in such demand, and why the most wanted kind is first-person video of ordinary work in real workplaces. Our guide to egocentric data collection describes how it is recorded.

If your crews do hands-on work and you wonder whether recordings of it would be useful, tell us about your business. CorpusMarket checks the fit and, where there is one, introduces you to a recording programme run by our partner Amelia Robotics.

Questions and answers

Frequently asked questions

What is imitation learning in robotics?

A way of teaching a robot by showing it examples of a task being done. The robot's model learns to produce the same kind of motion when it sees a similar situation. The phrase is also used in child development, which is a different subject.

What is the difference between imitation learning and behavior cloning?

Behavior cloning is the simplest form of imitation learning. It trains a model to output the demonstrated action for each recorded moment. Imitation learning is the wider family, which also includes methods that correct the robot when it drifts.

What is a diffusion policy?

A robot controller that produces motion by starting from random noise and refining it, step by step, into a short sequence of actions that fits what the camera sees. It copes well with tasks that have more than one right way to do them.

What does sim2real mean?

Short for simulation to reality. It is the step of taking a skill a robot learned in a simulator and getting it to work on a real robot, where friction, lighting and materials differ from the simulated version.

Do robots learn from watching videos of people?

Increasingly, yes. Video of people working is used to give large models a general sense of how tasks unfold, and robot recordings then connect that to the robot's own body. The video has to show the hands and the whole task to be useful.

Next step

Tell us about your business

If your crew does hands-on work, a robotics lab may pay for video of it. Send your details and we will tell you whether it fits and what happens next.

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days

CorpusMarket is independent of Amelia Robotics and may receive a referral fee if your business joins their programme. Nothing on this page is a promise of income.