# Egocentric data collection

> Egocentric data is video recorded from the point of view of the person doing the task, with the camera worn on the head and the hands in view. Robotics labs want a great deal of it, and most of it has to be recorded on purpose, in the places where the work is done.

By [Linus Näslund](https://corpusmarket.com/about.md), Co-Founder

## What is egocentric data

Egocentric means "from the self". Egocentric data is a recording made from the point of view of the person doing a task. The camera is worn, usually on the head, and it points where the person is looking or down at the hands. The viewer sees the workbench, the tool and two hands, and does not see the worker's face.

Egocentric data collection is the work of producing those recordings in an organized way: deciding which tasks to record, getting cameras onto people, checking what comes back, and labeling it so a lab can use it. You will also see it called first-person video, POV data for robotics, or egocentric video data. They all mean the same thing.

The opposite is exocentric or third-person video, filmed from outside by a camera on a tripod, a wall or a bystander's phone.

## Why robot learning wants the first-person view

A robot that works with its hands has cameras on its own body, often at head height and sometimes at the wrists. What it sees while working is close to what a person sees while working: a surface, some objects, and its own grippers reaching in from the bottom of the frame. Video recorded from a person's head is the closest match a human can give to that view.

There are three practical reasons labs ask for it.

1. **The hands stay visible.** From the outside, the worker's body blocks the work half the time. From the head, the hands and the object are in the middle of the picture.
2. **It shows where attention goes.** The head turns toward the next thing the worker needs. That is a cue about what matters in the scene.
3. **It can be recorded anywhere.** A worn camera goes wherever the worker goes. Nobody has to install anything.

The limit should be stated just as plainly. A video of human hands does not tell a robot which motors to turn. Turning human video into something a robot can act on is an open research problem, and labs combine it with data recorded on robots. Our comparison of [teleoperation and egocentric video](https://corpusmarket.com/robotics/teleoperation-vs-egocentric-video.md) covers that trade-off, and [how robots learn](https://corpusmarket.com/robotics/how-robots-learn.md) explains what the video is used for.

## How collection works in practice

The details differ between programmes, but the parts are the same almost everywhere.

| Part | What it is |
|---|---|
| Mount | A headband, cap or hardhat mount that holds the camera steady and leaves both hands free |
| Camera | An action camera, camera glasses, a headset, or an ordinary phone |
| Capture app | Starts and stops a recording, ties it to a task, and uploads it |
| Task list | The tasks the lab wants, each with a clear start and finish |
| Acceptance criteria | The rules a recording has to meet to be approved |
| Metadata | The facts stored with the video: which task, what kind of place, sometimes sensor readings |

A typical session is short. The worker puts on the mount, picks a task in the app, starts recording, does the task the way they normally would, and stops. The recording uploads, and a reviewer or an automated check decides whether it passes.

### What reviewers look for

Acceptance criteria are written by whoever pays for the data, so they vary. The common ones are that the hands and the work stay in frame, that the task runs from start to finish without cuts, that the picture is sharp and well lit enough to follow, and that the work is done at a natural pace. Staged or slowed-down work is usually rejected, because it teaches a model something that is not true of the real job.

### What is stored with the video

Video alone is hard to search and hard to learn from. Recordings are usually stored with a task name, a short description of what happens, and basic facts about the setting. Some programmes add the phone's motion sensor data. Later, a lab may add its own labels, such as where the hands are in each frame or when one step ends and the next begins.

**[Tell us about your business](https://corpusmarket.com/robotics)** – Two minutes, no commitment. We contact you within 2 business days.

## What makes egocentric data collection hard

**Bystanders and privacy.** A head-mounted camera pointed at the hands still catches whatever else is in view: a co-worker walking past, a customer, a screen, a document on the bench. The people recording agreed to it. The people around them may not have. How that is handled depends on the workplace and the programme, and it is one of the first things discussed before a business starts. Our page on [recording at work](https://corpusmarket.com/robotics/recording-at-work.md) describes the questions that usually come up.

**Consistency.** A hundred people will wear the same mount a hundred ways. Cameras tilt, lenses get dusty, someone forgets to press stop. Much of the effort in a collection programme goes into simple, boring things: a mount that sits the same way every time, an app that is hard to use wrong, and quick feedback when a recording fails review.

**Rare tasks.** Common tasks pile up fast. A kitchen produces hours of chopping every day. The tasks a robot most needs help with are often the ones that happen once a week, such as clearing a jam, fixing a mistake or handling an odd-shaped part. Collecting those takes patience and a task list that asks for them by name.

**Variety.** Many hours from one room teach a model that room. Labs want the same task across many sites, tools, lighting conditions and people, which is why they look beyond their own buildings.

## How it differs from third-person and fixed-camera video

| | Egocentric | Third-person | Fixed camera |
|---|---|---|---|
| Where the camera is | On the worker's head | Held or placed nearby | On a wall or ceiling |
| What it shows best | Hands, tools, the object | The whole body and posture | The room and who is in it |
| Hands in view | Nearly always | Often blocked by the body | Small and far away |
| Moves with the work | Yes | Only if someone moves it | No |
| Who starts it | The worker | A camera operator | Runs all the time |

This is why existing security footage is rarely what a robotics lab asks for. It was recorded for a different purpose, from a viewpoint that shows the room and hides the hands. Some research datasets record both views at once so that models can learn to relate them. The guide to [egocentric video datasets](https://corpusmarket.com/robotics/egocentric-video-datasets.md) lists the public ones.

## Who collects egocentric data

- **Research labs** record it themselves, usually in small amounts and for a specific study.
- **Robotics companies** run in-house collection in their own facilities and in mock-ups of homes and workplaces.
- **Data vendors** hire and manage people to record tasks to a specification.
- **Contributor apps** pay individuals to record everyday tasks at home.
- **Business programmes** pay a company for recordings made by its own crew during real work.

The last group is the newest, and it exists because real workplaces are where the missing variety is. A page for buyers, [robotics data providers](https://corpusmarket.com/robotics/robotics-data-providers.md), compares these sources from the lab's side.

## How a business takes part

A business does not need cameras, technical staff or any knowledge of robotics. It needs people who do hands-on work and are willing to record it.

CorpusMarket is an independent broker. We take enquiries from businesses, check whether the work fits, and introduce them to a recording programme run by our partner Amelia Robotics. In that programme a worker wears a phone on a hardhat or headband mount with the camera pointing down at the hands, and records agreed tasks in the programme's app. Recording is voluntary for each worker. The business is paid for the recording hours that pass the programme's quality review, and how that is shared with the workers is set by the business together with the programme.

The steps are laid out in [how it works](https://corpusmarket.com/robotics/how-it-works.md), and the kinds of work the programme covers are listed under [industries](https://corpusmarket.com/robotics/industries.md). If you want to know whether your work fits, [tell us about your business](https://corpusmarket.com/robotics.md#form). It takes two minutes, we reply within 2 business days, and nothing is shared outside CorpusMarket before we have spoken with you.

## Frequently asked questions

### What is egocentric data?

Recordings made from the point of view of the person doing a task. In practice it is video from a camera worn on the head or chest, with the hands and the work in view, sometimes with extra signals such as motion sensor readings or a short description of the task.

### What is the difference between egocentric and exocentric video?

Egocentric video is filmed from where the person's eyes are. Exocentric video is filmed from the outside, by a camera on a tripod, a wall or another person. The first shows what the worker sees, the second shows the worker.

### What equipment is used for egocentric video data?

Research projects have used head-mounted action cameras, camera glasses and headsets. Programmes that collect at scale often use an ordinary phone on a head mount, because workers already know how to use one and it needs no special hardware.

### Is POV data for robotics the same as egocentric data?

Yes. POV, first-person and egocentric all describe the same thing, a recording from the viewpoint of the person doing the work. Egocentric is the term used in research papers.

### Can a business take part in egocentric data collection?

Yes, if its people do hands-on work and are willing to record it. The business agrees a task list with a recording programme, workers who choose to take part record those tasks, and the business is paid for the recording hours that pass review.
