CorpusMarket Robotics

Robot teleoperation data vs egocentric video

There are two ways to record real-world examples for a robot. A person can steer the robot through the task, or a person can do the task and film it from their own point of view. Each gives the model something the other cannot, which is why most labs use both.

By Linus Näslund, Co-Founder

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days
Contents07
  1. 01What robot teleoperation data is
  2. 02Common teleoperation rigs
  3. 03What egocentric video is
  4. 04Teleoperation and egocentric video compared
  5. 05What human video is missing
  6. 06Why labs use both
  7. 07Where each fits

What robot teleoperation data is

Teleoperation means operating a machine from a distance. In robot learning it has a narrower meaning: a person steers the robot through a task, in real time, while everything is recorded. The person supplies the skill. The robot supplies the body.

Robot teleoperation data is what that session leaves behind. It has two halves that are saved together, many times a second:

  • What the robot sensed. Images from its cameras, the position of each joint, sometimes force or touch readings.
  • What the robot did. The commands sent to its motors, or the path its gripper followed.

That pairing is the reason labs value it. A model trained on teleoperation data sees an image and learns the action that went with it, in the exact body it will later control. Nothing has to be translated.

Common teleoperation rigs

Teleoperated robots are steered in a handful of ways. The names differ between labs, but the rigs fall into four groups.

VR controllers. The operator holds the hand controllers from a consumer virtual reality headset. The robot's gripper follows the controller's position and rotation, and a trigger opens and closes it.

Leader-follower arms. The operator moves a small, light copy of the robot arm by hand. The real arm mirrors every joint. It feels natural for two-handed work, and the copy gives the operator a physical sense of the robot's reach.

Exoskeleton gloves and motion capture. The operator wears a glove or an arm frame that measures finger and joint angles, and a robot hand with fingers follows. This is used for dexterous hands where a simple trigger is not enough.

Handheld grippers with a camera. The operator holds a gripper shaped like the robot's own, with a camera mounted on it, and does the task directly with no robot present. Strictly speaking this is not teleoperation, since no robot is being steered. It sits between the two worlds: the data looks like the robot's wrist camera, and it can be recorded anywhere.

What egocentric video is

Egocentric video is recorded by the person doing the task, from a camera worn on the head, with the hands in view. There is no robot involved. A worker does the job the normal way and the camera records what they see. Our guide to egocentric data collection covers mounts, task lists and review in detail.

Teleoperation and egocentric video compared

TeleoperationEgocentric video
Embodiment matchExact. The data is recorded in the robot's own bodyNone. Human hands, human arms, human height
Contains robot actionsYes, every motor commandNo. Actions have to be inferred
Cost per hourHigh. A robot, a trained operator and a prepared space for every hourLow. A worn camera and a person already doing the work
ScalabilityLimited by the number of robots and operatorsLimited mainly by how many people are willing to record
Variety of placesNarrow. Robots are hard to move, so most data comes from labs and staged roomsWide. A worn camera goes into real kitchens, shops and job sites
Speed of the workSlower than a person, with pauses and correctionsNatural working pace
What the model has to bridgeLittle. Mostly the gap between the recorded rooms and new onesA lot. From a human hand to a gripper, and from watching to acting

No row in that table settles the question alone. Teleoperation is precise and narrow. Egocentric video is broad and indirect.

What human video is missing

It is worth being exact about the weakness of egocentric video, because it is easy to oversell.

A first-person recording shows a pair of hands opening a valve. It does not say how far any joint turned, how hard the hand gripped, or what a two-finger gripper should do in its place. A robot's body is different from a person's in size, reach, strength and number of fingers. The camera sits in a different spot. None of the robot's own signals exist in the recording.

Researchers handle this in several ways, and none is finished work:

  • Estimating hand position in each frame, so the video gains something like an action track.
  • Retargeting, which maps a human hand motion onto a robot's gripper.
  • Pretraining, where a model first learns from large amounts of human video what objects are and how tasks unfold, and is then trained on a smaller amount of robot data to learn the actual movements.

In every one of these, some robot data is still needed. Human video reduces how much. It does not remove the need.

Why labs use both

The two sources cover each other's gaps. Teleoperation gives a model correct actions in the right body, in a small number of places. Egocentric video gives it a wide view of how work is done across many places, people and tools, with no actions attached.

A common pattern is to train broadly on human video and other large sources, then fine-tune on teleoperation data for the specific robot. The human video supplies the variety. The teleoperation data supplies the grounding. How robots learn describes that process step by step, and robot training data sets both sources next to simulation and the robot's own practice.

Published data follows the same split. Some public collections are recorded on robots by teleoperation, others are first-person video of people. The page on egocentric video datasets lists both kinds.

Where each fits

Teleoperation fits when a lab is teaching one robot a defined set of tasks and needs the result to be reliable. It also fits the last stage of almost any project, when a general model is adapted to a particular machine.

Egocentric video fits when the goal is breadth: many kinds of work, many settings, the ordinary variation that a lab cannot stage. It is also the only one of the two that people outside robotics can supply. A restaurant or a plumbing company has no robot to steer, but its people do skilled work with their hands all day.

That is the part CorpusMarket is involved in. We are an independent broker. We take enquiries from businesses whose crews could record their work and introduce those that fit to a recording programme run by our partner Amelia Robotics. In that programme a worker wears a phone on a head mount, records agreed tasks, and the business is paid for the recording hours that pass review. The kinds of work covered are listed under industries.

If you run a business with hands-on work, tell us about your business. If you are on the lab side and need either kind of data captured, the page for robotics labs explains how to describe what you need.

Questions and answers

Frequently asked questions

What is robot teleoperation?

A person controls a robot remotely, in real time, and the robot copies the person's movements. When it is done to produce training examples, the robot's camera images and its own movements are saved together.

Are teleoperated robots autonomous?

No. A teleoperated robot is being steered by a person. Teleoperation is used to record examples for training and, in some products, to let a person take over when the robot cannot manage a task alone.

Why is teleoperation data collection expensive?

Every hour of data needs a robot, a trained operator and a place set up for the task, and the robot usually works more slowly than a person would. The cost grows in step with the hours recorded.

Can a robot learn from human video alone?

Not reliably today. Human video shows what happens but contains no robot actions, so models trained on it are generally also trained or adjusted on data recorded with a robot.

Which kind of data can a business supply?

Egocentric video. Teleoperation needs the lab's robot and trained operators. First-person video needs a worn camera and people who do the work, which is what a business already has.

Next step

Tell us about your business

If your crew does hands-on work, a robotics lab may pay for video of it. Send your details and we will tell you whether it fits and what happens next.

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days

CorpusMarket is independent of Amelia Robotics and may receive a referral fee if your business joins their programme. Nothing on this page is a promise of income.