About two minutes · no cost · no commitment
CorpusMarket Robotics
Egocentric video datasets
A small number of public datasets come up in almost every paper on robots learning from video. Some are first-person recordings of people. Others are recordings made on robots. This page lists the well-known ones, says what each contains, and explains why labs still pay for new footage.
By Linus Näslund, Co-Founder
- No cost to apply
- Nothing shared before we talk
- Reply within 2 business days
Contents06
Two kinds of dataset
An egocentric video dataset is a published collection of first-person recordings of people doing things, with labels attached. A robot dataset is a published collection of recordings made on robots, with the robot's camera images and the actions it took.
They are often mentioned together, and they answer different needs. Human video is broad and contains no robot actions. Robot data contains actions and covers far fewer places. Teleoperation versus egocentric video explains the trade-off. The figures below are quoted as each project states them on its own site.
Human egocentric video datasets
| Dataset | Released by | What it contains | Viewpoint | Access |
|---|---|---|---|---|
| Ego4D | A consortium of 13 universities in partnership with Facebook AI | 3,670 hours of daily-life activity video from 74 locations in 9 countries, with benchmark tasks | First-person, worn camera | License agreement accepted before download |
| Ego-Exo4D | University groups together with FAIR, Meta's AI research group | 1,286.3 hours of skilled activities such as cooking and bike repair, in 5,035 takes | First-person camera glasses plus four to five stationary outside cameras, recorded at once | Separate license signed before download |
| EPIC-KITCHENS-100 | University of Bristol and University of Catania | 100 hours, 20M frames and 90K action segments of kitchen activity in 45 kitchens | First-person, head-mounted | Creative Commons Attribution-NonCommercial 4.0 |
| HOI4D | Tsinghua University, Peking University and Shanghai Qi Zhi Institute | 2.4M RGB-D egocentric video frames over 4000 sequences of people handling objects indoors | First-person, color plus depth | CC BY-NC 4.0 |
| EgoDex | Apple | 829 hours of tabletop manipulation across 194 tasks, with 3D pose of the head, upper body and hands | First-person, recorded on a headset | CC BY-NC-ND |
Ego4D is the broadest of these. It was built to cover everyday life in general: household, outdoor, workplace and leisure scenes. Its benchmarks test things such as remembering where an object was left and predicting what a person will do next.
Ego-Exo4D adds a second view. Each take is filmed from the person's own camera glasses and from several fixed cameras around them at the same moment. That makes it useful for studying how the first-person view relates to the outside view.
EPIC-KITCHENS is narrower. People wore a head-mounted camera while cooking and cleaning up in their own kitchens, and the recordings come with narrations of what is being done. The setting is one kind of room, recorded in depth.
HOI4D and EgoDex focus on the hands. Both add three-dimensional information about hand position, which is the piece that ordinary video lacks when the goal is to teach a robot to manipulate objects. EgoDex covers tabletop tasks only.
Robot manipulation datasets
These are not egocentric video in the usual sense. No person appears doing the task with their own hands. A robot does it, usually steered by a person, and the robot's cameras and movements are recorded.
| Dataset | Released by | What it contains | Viewpoint | Access |
|---|---|---|---|---|
| Open X-Embodiment | The Open X-Embodiment Collaboration, with code hosted by Google DeepMind | 1M+ real robot trajectories spanning 22 robot embodiments, pooled from 60 existing robot datasets | Robot cameras, varying by source dataset | Open download |
| DROID | The DROID Dataset Team, across 13 institutions | 76k demonstration trajectories or 350h of interaction data in 564 scenes, on a Franka Panda arm steered with VR controllers | Two external stereo cameras and one wrist camera | Open download |
| BridgeData V2 | Walke and colleagues (research paper, 2023) | 60,096 trajectories in 24 environments on a WidowX 250 arm, mostly teleoperated demonstrations | Robot setup | Creative Commons Attribution 4.0 |
Open X-Embodiment is a pooling effort. Research labs contributed datasets they had already recorded on their own robots, and the collaboration pooled them. Its value is the range of robot bodies, from single arms to two-armed robots and quadrupeds.
DROID was recorded with the same hardware setup at every participating institution. The arm sits on a portable desk, which let the team record in many different rooms. Robot data is otherwise often tied to a fixed lab bench.
BridgeData V2 was recorded on one model of arm. Most of its trajectories are demonstrations in which a person steered the arm with a VR controller, and the rest come from a scripted pick-and-place routine.
What public datasets are good for
Public datasets made much of the current research possible. They are used in three main ways.
- Benchmarks. Everyone tests on the same data, so results can be compared.
- Pretraining. A model learns general things about objects, hands and actions from a large public set before it is trained on something narrower. How robots learn describes that sequence.
- Method development. A new idea can be tried on existing data before anyone pays to collect more.
Their limits for commercial training
Each dataset is published under its own terms, and the terms differ. The table shows the spread. EPIC-KITCHENS-100, HOI4D and EgoDex carry Creative Commons licenses with a non-commercial condition, and the EPIC-KITCHENS site gives a contact for commercial licensing. Ego4D and Ego-Exo4D are released under their own license agreements, which are signed before access is granted. BridgeData V2 is under an attribution license with no non-commercial condition.
Academic and non-commercial terms are common among the human video datasets. A company building a product reads each license for itself. This page describes the stated terms and is not a view on what any license permits in a particular case.
There are other limits that have nothing to do with licensing:
- Coverage. The human datasets lean toward homes, kitchens and leisure. Trades, job sites, stockrooms and commercial kitchens during service are thin or absent.
- Format. Each dataset was recorded with its own camera, mount and labeling scheme, which may not match what a given robot needs.
- Shared by everyone. Every lab has the same public data. It cannot be what sets one model apart.
Why labs commission new collection anyway
For those reasons, labs that build products pay for new footage, recorded to their own specification and under terms written for commercial use. They ask for particular tasks, a particular camera position, and settings the public data does not reach. Egocentric data collection describes how that work is organized, and video data licensing covers how rights to footage are usually arranged.
Real workplaces are where most of the missing variety is, and they belong to businesses. CorpusMarket is an independent broker that takes enquiries from businesses whose crews could record their work and introduces those that fit to a recording programme run by our partner Amelia Robotics. If that could be your business, tell us about your business. Labs looking for footage can start at the page for robotics labs.
Questions and answers
Frequently asked questions
What is an egocentric video dataset?
A collection of videos recorded from the point of view of the person doing the activity, usually from a head-mounted camera, published together with labels such as descriptions of the actions or the position of the hands.
What is Ego4D?
Ego4D is a large egocentric dataset and benchmark suite released by a consortium of 13 universities in partnership with Facebook AI. Its site describes 3,670 hours of daily-life activity video recorded in 74 locations across 9 countries.
What is EPIC-KITCHENS?
A first-person dataset of people cooking and doing kitchen tasks in their own homes, from the University of Bristol and the University of Catania. EPIC-KITCHENS-100 has 100 hours of recording and is published under a non-commercial Creative Commons license.
Is Open X-Embodiment an egocentric dataset?
No. Open X-Embodiment is a robot dataset. It pools recordings made on many different robots, so it contains the robots' camera images and actions, not video of people working.
Can public robotics datasets be used to train a commercial product?
It depends on the dataset. Each one is published under its own terms, and several of the best-known egocentric datasets carry non-commercial licenses. The terms are stated by each project and are read dataset by dataset.
More pages
Read next
Egocentric data collection
Egocentric data is video recorded from the point of view of the person doing the task, with the camera worn on the head and the hands in view. Robotics labs want a great deal of it, and most of it has to be recorded on purpose, in the places where the work is done.
Read moreRobotics training data
A language model learned from text that was already on the internet. A robot has no such archive to learn from. Robotics training data has to be made on purpose, and most of the cost and effort in teaching a robot goes into getting enough of it.
Read moreVideo data licensing for AI training
AI developers license video from businesses the same way they license text and images. Whether your video is worth licensing depends on what it shows, who is in it and whether you hold the rights. This page separates the two routes, licensing footage you already have and recording new footage that a lab has asked for.
Read moreNext step
Tell us about your business
If your crew does hands-on work, a robotics lab may pay for video of it. Send your details and we will tell you whether it fits and what happens next.
- No cost to apply
- Nothing shared before we talk
- Reply within 2 business days