# Robot teleoperation data vs egocentric video

> There are two ways to record real-world examples for a robot. A person can steer the robot through the task, or a person can do the task and film it from their own point of view. Each gives the model something the other cannot, which is why most labs use both.

By [Linus Näslund](https://corpusmarket.com/about.md), Co-Founder

## What robot teleoperation data is

Teleoperation means operating a machine from a distance. In robot learning it has a narrower meaning: a person steers the robot through a task, in real time, while everything is recorded. The person supplies the skill. The robot supplies the body.

Robot teleoperation data is what that session leaves behind. It has two halves that are saved together, many times a second:

- **What the robot sensed.** Images from its cameras, the position of each joint, sometimes force or touch readings.
- **What the robot did.** The commands sent to its motors, or the path its gripper followed.

That pairing is the reason labs value it. A model trained on teleoperation data sees an image and learns the action that went with it, in the exact body it will later control. Nothing has to be translated.

## Common teleoperation rigs

Teleoperated robots are steered in a handful of ways. The names differ between labs, but the rigs fall into four groups.

**VR controllers.** The operator holds the hand controllers from a consumer virtual reality headset. The robot's gripper follows the controller's position and rotation, and a trigger opens and closes it.

**Leader-follower arms.** The operator moves a small, light copy of the robot arm by hand. The real arm mirrors every joint. It feels natural for two-handed work, and the copy gives the operator a physical sense of the robot's reach.

**Exoskeleton gloves and motion capture.** The operator wears a glove or an arm frame that measures finger and joint angles, and a robot hand with fingers follows. This is used for dexterous hands where a simple trigger is not enough.

**Handheld grippers with a camera.** The operator holds a gripper shaped like the robot's own, with a camera mounted on it, and does the task directly with no robot present. Strictly speaking this is not teleoperation, since no robot is being steered. It sits between the two worlds: the data looks like the robot's wrist camera, and it can be recorded anywhere.

## What egocentric video is

Egocentric video is recorded by the person doing the task, from a camera worn on the head, with the hands in view. There is no robot involved. A worker does the job the normal way and the camera records what they see. Our guide to [egocentric data collection](https://corpusmarket.com/robotics/egocentric-data-collection.md) covers mounts, task lists and review in detail.

**[Tell us about your business](https://corpusmarket.com/robotics)** – Two minutes, no commitment. We contact you within 2 business days.

## Teleoperation and egocentric video compared

| | Teleoperation | Egocentric video |
|---|---|---|
| Embodiment match | Exact. The data is recorded in the robot's own body | None. Human hands, human arms, human height |
| Contains robot actions | Yes, every motor command | No. Actions have to be inferred |
| Cost per hour | High. A robot, a trained operator and a prepared space for every hour | Low. A worn camera and a person already doing the work |
| Scalability | Limited by the number of robots and operators | Limited mainly by how many people are willing to record |
| Variety of places | Narrow. Robots are hard to move, so most data comes from labs and staged rooms | Wide. A worn camera goes into real kitchens, shops and job sites |
| Speed of the work | Slower than a person, with pauses and corrections | Natural working pace |
| What the model has to bridge | Little. Mostly the gap between the recorded rooms and new ones | A lot. From a human hand to a gripper, and from watching to acting |

No row in that table settles the question alone. Teleoperation is precise and narrow. Egocentric video is broad and indirect.

## What human video is missing

It is worth being exact about the weakness of egocentric video, because it is easy to oversell.

A first-person recording shows a pair of hands opening a valve. It does not say how far any joint turned, how hard the hand gripped, or what a two-finger gripper should do in its place. A robot's body is different from a person's in size, reach, strength and number of fingers. The camera sits in a different spot. None of the robot's own signals exist in the recording.

Researchers handle this in several ways, and none is finished work:

- **Estimating hand position** in each frame, so the video gains something like an action track.
- **Retargeting**, which maps a human hand motion onto a robot's gripper.
- **Pretraining**, where a model first learns from large amounts of human video what objects are and how tasks unfold, and is then trained on a smaller amount of robot data to learn the actual movements.

In every one of these, some robot data is still needed. Human video reduces how much. It does not remove the need.

## Why labs use both

The two sources cover each other's gaps. Teleoperation gives a model correct actions in the right body, in a small number of places. Egocentric video gives it a wide view of how work is done across many places, people and tools, with no actions attached.

A common pattern is to train broadly on human video and other large sources, then fine-tune on teleoperation data for the specific robot. The human video supplies the variety. The teleoperation data supplies the grounding. [How robots learn](https://corpusmarket.com/robotics/how-robots-learn.md) describes that process step by step, and [robot training data](https://corpusmarket.com/robotics/robot-training-data.md) sets both sources next to simulation and the robot's own practice.

Published data follows the same split. Some public collections are recorded on robots by teleoperation, others are first-person video of people. The page on [egocentric video datasets](https://corpusmarket.com/robotics/egocentric-video-datasets.md) lists both kinds.

## Where each fits

Teleoperation fits when a lab is teaching one robot a defined set of tasks and needs the result to be reliable. It also fits the last stage of almost any project, when a general model is adapted to a particular machine.

Egocentric video fits when the goal is breadth: many kinds of work, many settings, the ordinary variation that a lab cannot stage. It is also the only one of the two that people outside robotics can supply. A restaurant or a plumbing company has no robot to steer, but its people do skilled work with their hands all day.

That is the part CorpusMarket is involved in. We are an independent broker. We take enquiries from businesses whose crews could record their work and introduce those that fit to a recording programme run by our partner Amelia Robotics. In that programme a worker wears a phone on a head mount, records agreed tasks, and the business is paid for the recording hours that pass review. The kinds of work covered are listed under [industries](https://corpusmarket.com/robotics/industries.md).

If you run a business with hands-on work, [tell us about your business](https://corpusmarket.com/robotics.md#form). If you are on the lab side and need either kind of data captured, the page [for robotics labs](https://corpusmarket.com/robotics/for-labs.md) explains how to describe what you need.

## Frequently asked questions

### What is robot teleoperation?

A person controls a robot remotely, in real time, and the robot copies the person's movements. When it is done to produce training examples, the robot's camera images and its own movements are saved together.

### Are teleoperated robots autonomous?

No. A teleoperated robot is being steered by a person. Teleoperation is used to record examples for training and, in some products, to let a person take over when the robot cannot manage a task alone.

### Why is teleoperation data collection expensive?

Every hour of data needs a robot, a trained operator and a place set up for the task, and the robot usually works more slowly than a person would. The cost grows in step with the hours recorded.

### Can a robot learn from human video alone?

Not reliably today. Human video shows what happens but contains no robot actions, so models trained on it are generally also trained or adjusted on data recorded with a robot.

### Which kind of data can a business supply?

Egocentric video. Teleoperation needs the lab's robot and trained operators. First-person video needs a worn camera and people who do the work, which is what a business already has.
