# Robotics training data

> A language model learned from text that was already on the internet. A robot has no such archive to learn from. Robotics training data has to be made on purpose, and most of the cost and effort in teaching a robot goes into getting enough of it.

By [Linus Näslund](https://corpusmarket.com/about.md), Co-Founder

## What robotics training data is

Robotics training data is any recording that helps a robot learn to act in the physical world. It pairs what was seen with what was done. For a person folding a towel, that is the view of the towel and the motion of the hands. For a robot arm, it is the camera image and the position of every joint, many times a second.

The reason it gets so much attention is simple. The models that drive today's robots are trained the same way as the models that write text: show them a very large number of examples. Text had the internet. Physical work has nothing like it, so the examples have to be recorded.

## The four main kinds

| Kind | How it is made | Strength | Limit |
|---|---|---|---|
| Human video | People record themselves doing tasks, usually from a head-mounted camera | Cheap to scale, real places and real variety | The robot has to translate a human hand into its own |
| Teleoperation | A person steers a robot through the task | Recorded in the robot's own body | Slow and costly: one operator, one robot, one hour |
| Simulation | Tasks are generated in a virtual world | Unlimited volume | Simulated physics differs from the real thing |
| Robot experience | The robot tries, fails and is corrected | Exactly the right data | Needs a robot that already half works |

Labs mix all four. The mix is shifting toward human video, because it is the only kind that can reach the volume and variety the models seem to need. [Teleoperation versus egocentric video](https://corpusmarket.com/robotics/teleoperation-vs-egocentric-video.md) compares the two real-world sources in detail.

## Why real work is the scarce part

A lab can rent a warehouse and pay people to fold laundry all day. Many do. The trouble is that the result shows one room, one set of objects and a handful of people. A robot trained on it does well in that room.

What labs lack is footage from the places work really happens: a commercial kitchen during service, a mechanic's bay, a job site, a stockroom. Those places belong to businesses, and the people who know the work are their employees. That is why some labs and data-collection programmes now pay businesses for recordings made by their own crews.

The preferred format is first-person video, with the camera worn on the head and the hands in view. It is called egocentric video, and our guide to [egocentric data collection](https://corpusmarket.com/robotics/egocentric-data-collection.md) explains how it is recorded and checked.

**[Tell us about your business](https://corpusmarket.com/robotics)** – Two minutes, no commitment. We contact you within 2 business days.

## What makes a recording useful

Labs review recordings before they pay for them. The usual criteria are:

- **The hands and the work stay in frame** for the whole task.
- **The task is complete**, from picking up the first tool to putting down the last.
- **The pace is natural.** Slowed-down or staged work teaches the wrong thing.
- **The setting varies.** Different sites, lighting, tools and people.
- **The footage can be used.** The people in it agreed, and nothing in frame is off limits.

The last point is a matter for each workplace. What may be filmed where differs between businesses and between jobs, which is why it is gone through before recording starts. See [recording at work](https://corpusmarket.com/robotics/recording-at-work.md).

## What happens to the data after it is recorded

A recording is not fed to a robot as it is. Between the phone and the model there are usually several steps, and they explain why labs are particular about what they accept.

1. **Review.** Each recording is checked against the criteria above, by people, by software or both. Recordings that fail are not paid for.
2. **Labelling.** The task is named and often split into steps, such as "pick up the trowel" or "spread the mortar". Some labs also extract the position of the hands in each frame.
3. **Mixing.** The human video is combined with robot recordings and simulated data, so the model sees the same kind of task from several sources.
4. **Training and testing.** The model is trained, then tried on a real robot, usually on tasks and in places it has not seen. Where it fails tells the lab what to record next.

That last step is why demand keeps moving. A lab that has plenty of kitchen footage may need stockrooms next, and a model that handles rigid objects may still fail on cables and cloth. For a business, it means the work a lab asks for today is not a fixed list.

## Why this is robot learning data and not ordinary video

The same hour of footage can be worth a great deal to a robotics lab and nothing to anyone else. What sets robot learning data apart is that it is recorded for one purpose: every choice, from where the camera sits to which tasks are picked, is made so that a model can connect what it sees to what the hands do. Our guide to [how robots learn](https://corpusmarket.com/robotics/how-robots-learn.md) explains that connection, and [VLA models](https://corpusmarket.com/robotics/vla-models.md) covers the kind of model most of this data now goes into.

## Where businesses fit in

A business does not need to know anything about robotics to supply this data. It needs people who do hands-on work and are willing to record it. The kinds of work labs ask for today include the trades, [construction](https://corpusmarket.com/robotics/construction.md), auto service, warehousing, kitchens, cleaning, manufacturing, retail, hotels and farming. The [industries](https://corpusmarket.com/robotics/industries.md) page lists them.

CorpusMarket takes enquiries from businesses, checks whether the work fits, and introduces them to a recording programme run by our partner Amelia Robotics. [How it works](https://corpusmarket.com/robotics/how-it-works.md) describes the steps.

## For labs looking for data

If you are on the buying side, [robotics data providers](https://corpusmarket.com/robotics/robotics-data-providers.md) compares the ways labs source real-world data, and the page [for robotics labs](https://corpusmarket.com/robotics/for-labs.md) explains how to describe what you need captured.

## Frequently asked questions

### What is robotics training data?

Recordings that show a robot how a physical task is done. The main kinds are video of people doing the task, recordings of a robot being steered through the task by a person, data generated in simulation, and the robot's own attempts.

### Why can't robots just learn from videos on the internet?

Internet video helps, but it is filmed from the outside, edited, and rarely shows a whole task from start to finish. A robot learns best from footage taken from where the worker's eyes are, with the hands in view the whole time, of ordinary tasks done at ordinary speed.

### Who supplies real-world robot learning data?

Labs record some themselves, hire data-collection companies for more, and increasingly pay businesses and individuals to record everyday work. Programmes that pay a business for its crew's recordings are a recent addition.

### How much data does a robot need?

There is no agreed number. Labs describe their needs in thousands to hundreds of thousands of hours, and the answer changes with the method and the range of tasks. What is consistent is that variety across places and people counts as much as volume.

### Can a business sell video it already has, such as security footage?

Usually not for this purpose. Fixed cameras show a room from above, not the hands doing the work. Labs ask for first-person recordings made for the task. Existing archives can have value for other kinds of AI, which is a separate question.
