Home / Blog / Egocentric data

Egocentric data collection for robotics and embodied AI

Why first-person video matters for models that act in the world, what the major public datasets teach about scale and annotation, and what a capture program needs to get right.

Latryn Research Team9 min read4 cited sources

Most video on the internet is filmed by someone watching. Models that have to act, such as robot policies, assistants in smart glasses and agents that follow a person through a task, need the other view: the one from the eyes of the person doing the work. That first-person view is called egocentric data, and collecting it well is a discipline of its own.

Why the first-person view

An egocentric clip puts the hands, the objects they touch and the direction of attention in the same frame. It captures the order of steps, the small corrections and the moments of hesitation that a third-person camera misses. For robotics and embodied AI, that is the signal that matters: what a skilled person actually does, from where they actually stand.

What the public datasets teach

Three research datasets show what large-scale egocentric collection looks like and where the effort goes.

Scale and diversity. Ego4D collected 3,670 hours of daily-life video from 931 unique camera wearers in 74 locations across 9 countries [1]. Its benchmarks span remembering the past, understanding the present (including hand and object manipulation) and forecasting what happens next [1]. The lesson: diversity of people and places is designed in, not hoped for.

Annotation density. EPIC-KITCHENS-100 contains 100 hours of video, 20M frames and 90K actions recorded in 45 kitchen environments [2]. Its authors built a new annotation pipeline that produced denser and more complete labels, with 54% more actions per minute than the earlier version [2]. The lesson: how you annotate decides what a dataset is worth.

Skill and expertise. Ego-Exo4D captured skilled human activity from 740 participants in 13 cities, 1,286 hours of video recorded from first-person and third-person cameras at the same time, with gaze, IMU and 3D point clouds [3]. It also includes commentary written by coaches and teachers about how well each activity was performed [3]. The lesson: for skilled tasks, expert judgement is part of the data.

From watching to doing

Egocentric video is no longer only for recognition models. EgoMimic recorded human demonstrations with Project Aria glasses, paired them with 3D hand tracking, and trained a single robot policy on human and robot data together [4]. The authors report that an additional hour of hand data was significantly more valuable than an additional hour of robot data [4]. Human first-person data is cheaper to collect than robot teleoperation, and it scales with people rather than hardware.

What a capture program needs

Footage alone is not a dataset. The programs that work decide these things before anyone presses record:

  • A capture protocol. Which tasks, which environments, how long each session runs, how the camera is worn and what counts as a usable clip.
  • Consent and privacy. Written consent from every contributor, clear rules for bystanders, screens and documents in view, and how clips are stored and handed over.
  • An annotation schema. Which labels you need: hand keypoints, object boxes and contact, action segments with start and end times, narration, or quality judgements. Write the edge cases down.
  • Agreement checks. Overlap a share of clips between annotators and measure agreement with chance-corrected statistics, as with any human labelling. Our note on expert and crowd annotation explains how.
  • Review before delivery. Reject clips that break the protocol before they are labelled, and review labels before they reach your training pipeline.

How Latryn runs egocentric programs

Latryn Capture follows the same loop as our other practices. We agree the tasks, devices, consent process and annotation schema with you in writing. A pilot batch calibrates contributors and annotators on the edge cases. Contributors then record real tasks with head-mounted cameras, trained reviewers segment and label each clip, and every clip passes three gates (automated screen, structured scoring and human sign-off) before delivery.

See the structure of a sample egocentric clip or scope a capture program.

Need data built to this standard?

Tell us the benchmark, task format or dataset you need. We agree scope and acceptance checks before any work starts.

Scope a program See sample tasks

References

  1. Grauman, K., et al. (2022). Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022. arXiv:2110.07058
  2. Damen, D., et al. (2022). Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. International Journal of Computer Vision. arXiv:2006.13256
  3. Grauman, K., et al. (2024). Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives. CVPR 2024. arXiv:2311.18259
  4. Kareer, S., et al. (2024). EgoMimic: Scaling Imitation Learning via Egocentric Video. arXiv:2410.24221

Keep reading

Evaluation

What makes an AI benchmark trustworthy

Contamination, unfair tests and vague task statements quietly break evaluations. Here is how to build benchmarks whose scores you can believe.

8 min read · 5 sourcesTraining data

RLHF and preference data, explained for teams that collect it

Reinforcement learning from human feedback is only as good as the comparisons behind it. What to collect, who should collect it, and how DPO changes the pipeline.

9 min read · 4 sourcesAgent environments

How to build agent environments that test real work

Docker images, terminal tasks and verifiers: a practical anatomy of the environments used to train and evaluate AI agents.

8 min read · 4 sources