CorpusMarket Robotics

Egocentric video datasets

A small number of public datasets come up in almost every paper on robots learning from video. Some are first-person recordings of people. Others are recordings made on robots. This page lists the well-known ones, says what each contains, and explains why labs still pay for new footage.

By Linus Näslund, Co-Founder

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days
Contents06
  1. 01Two kinds of dataset
  2. 02Human egocentric video datasets
  3. 03Robot manipulation datasets
  4. 04What public datasets are good for
  5. 05Their limits for commercial training
  6. 06Why labs commission new collection anyway

Two kinds of dataset

An egocentric video dataset is a published collection of first-person recordings of people doing things, with labels attached. A robot dataset is a published collection of recordings made on robots, with the robot's camera images and the actions it took.

They are often mentioned together, and they answer different needs. Human video is broad and contains no robot actions. Robot data contains actions and covers far fewer places. Teleoperation versus egocentric video explains the trade-off. The figures below are quoted as each project states them on its own site.

Human egocentric video datasets

DatasetReleased byWhat it containsViewpointAccess
Ego4DA consortium of 13 universities in partnership with Facebook AI3,670 hours of daily-life activity video from 74 locations in 9 countries, with benchmark tasksFirst-person, worn cameraLicense agreement accepted before download
Ego-Exo4DUniversity groups together with FAIR, Meta's AI research group1,286.3 hours of skilled activities such as cooking and bike repair, in 5,035 takesFirst-person camera glasses plus four to five stationary outside cameras, recorded at onceSeparate license signed before download
EPIC-KITCHENS-100University of Bristol and University of Catania100 hours, 20M frames and 90K action segments of kitchen activity in 45 kitchensFirst-person, head-mountedCreative Commons Attribution-NonCommercial 4.0
HOI4DTsinghua University, Peking University and Shanghai Qi Zhi Institute2.4M RGB-D egocentric video frames over 4000 sequences of people handling objects indoorsFirst-person, color plus depthCC BY-NC 4.0
EgoDexApple829 hours of tabletop manipulation across 194 tasks, with 3D pose of the head, upper body and handsFirst-person, recorded on a headsetCC BY-NC-ND

Ego4D is the broadest of these. It was built to cover everyday life in general: household, outdoor, workplace and leisure scenes. Its benchmarks test things such as remembering where an object was left and predicting what a person will do next.

Ego-Exo4D adds a second view. Each take is filmed from the person's own camera glasses and from several fixed cameras around them at the same moment. That makes it useful for studying how the first-person view relates to the outside view.

EPIC-KITCHENS is narrower. People wore a head-mounted camera while cooking and cleaning up in their own kitchens, and the recordings come with narrations of what is being done. The setting is one kind of room, recorded in depth.

HOI4D and EgoDex focus on the hands. Both add three-dimensional information about hand position, which is the piece that ordinary video lacks when the goal is to teach a robot to manipulate objects. EgoDex covers tabletop tasks only.

Robot manipulation datasets

These are not egocentric video in the usual sense. No person appears doing the task with their own hands. A robot does it, usually steered by a person, and the robot's cameras and movements are recorded.

DatasetReleased byWhat it containsViewpointAccess
Open X-EmbodimentThe Open X-Embodiment Collaboration, with code hosted by Google DeepMind1M+ real robot trajectories spanning 22 robot embodiments, pooled from 60 existing robot datasetsRobot cameras, varying by source datasetOpen download
DROIDThe DROID Dataset Team, across 13 institutions76k demonstration trajectories or 350h of interaction data in 564 scenes, on a Franka Panda arm steered with VR controllersTwo external stereo cameras and one wrist cameraOpen download
BridgeData V2Walke and colleagues (research paper, 2023)60,096 trajectories in 24 environments on a WidowX 250 arm, mostly teleoperated demonstrationsRobot setupCreative Commons Attribution 4.0

Open X-Embodiment is a pooling effort. Research labs contributed datasets they had already recorded on their own robots, and the collaboration pooled them. Its value is the range of robot bodies, from single arms to two-armed robots and quadrupeds.

DROID was recorded with the same hardware setup at every participating institution. The arm sits on a portable desk, which let the team record in many different rooms. Robot data is otherwise often tied to a fixed lab bench.

BridgeData V2 was recorded on one model of arm. Most of its trajectories are demonstrations in which a person steered the arm with a VR controller, and the rest come from a scripted pick-and-place routine.

What public datasets are good for

Public datasets made much of the current research possible. They are used in three main ways.

  • Benchmarks. Everyone tests on the same data, so results can be compared.
  • Pretraining. A model learns general things about objects, hands and actions from a large public set before it is trained on something narrower. How robots learn describes that sequence.
  • Method development. A new idea can be tried on existing data before anyone pays to collect more.

Their limits for commercial training

Each dataset is published under its own terms, and the terms differ. The table shows the spread. EPIC-KITCHENS-100, HOI4D and EgoDex carry Creative Commons licenses with a non-commercial condition, and the EPIC-KITCHENS site gives a contact for commercial licensing. Ego4D and Ego-Exo4D are released under their own license agreements, which are signed before access is granted. BridgeData V2 is under an attribution license with no non-commercial condition.

Academic and non-commercial terms are common among the human video datasets. A company building a product reads each license for itself. This page describes the stated terms and is not a view on what any license permits in a particular case.

There are other limits that have nothing to do with licensing:

  • Coverage. The human datasets lean toward homes, kitchens and leisure. Trades, job sites, stockrooms and commercial kitchens during service are thin or absent.
  • Format. Each dataset was recorded with its own camera, mount and labeling scheme, which may not match what a given robot needs.
  • Shared by everyone. Every lab has the same public data. It cannot be what sets one model apart.

Why labs commission new collection anyway

For those reasons, labs that build products pay for new footage, recorded to their own specification and under terms written for commercial use. They ask for particular tasks, a particular camera position, and settings the public data does not reach. Egocentric data collection describes how that work is organized, and video data licensing covers how rights to footage are usually arranged.

Real workplaces are where most of the missing variety is, and they belong to businesses. CorpusMarket is an independent broker that takes enquiries from businesses whose crews could record their work and introduces those that fit to a recording programme run by our partner Amelia Robotics. If that could be your business, tell us about your business. Labs looking for footage can start at the page for robotics labs.

Questions and answers

Frequently asked questions

What is an egocentric video dataset?

A collection of videos recorded from the point of view of the person doing the activity, usually from a head-mounted camera, published together with labels such as descriptions of the actions or the position of the hands.

What is Ego4D?

Ego4D is a large egocentric dataset and benchmark suite released by a consortium of 13 universities in partnership with Facebook AI. Its site describes 3,670 hours of daily-life activity video recorded in 74 locations across 9 countries.

What is EPIC-KITCHENS?

A first-person dataset of people cooking and doing kitchen tasks in their own homes, from the University of Bristol and the University of Catania. EPIC-KITCHENS-100 has 100 hours of recording and is published under a non-commercial Creative Commons license.

Is Open X-Embodiment an egocentric dataset?

No. Open X-Embodiment is a robot dataset. It pools recordings made on many different robots, so it contains the robots' camera images and actions, not video of people working.

Can public robotics datasets be used to train a commercial product?

It depends on the dataset. Each one is published under its own terms, and several of the best-known egocentric datasets carry non-commercial licenses. The terms are stated by each project and are read dataset by dataset.

Next step

Tell us about your business

If your crew does hands-on work, a robotics lab may pay for video of it. Send your details and we will tell you whether it fits and what happens next.

  • No cost to apply
  • Nothing shared before we talk
  • Reply within 2 business days

CorpusMarket is independent of Amelia Robotics and may receive a referral fee if your business joins their programme. Nothing on this page is a promise of income.