What is egocentric (POV) video?

Egocentric vs third-person video

Most video is third-person. A camera on a tripod, a phone in someone else's hand or a security camera on the wall watches a person from outside. Egocentric video turns that around: the person doing the task wears the camera, so the frame moves with their head. It often shows what's in front of them, such as what their hands reach for and in what order. Our egocentric video data page shows how we collect this footage to order for AI teams.

Some datasets record both views at once. Ego-Exo4D captures egocentric and exocentric video of skilled activities at the same moment, so researchers can study the same action from inside and outside. It's a paired-view dataset, with first-person and outside footage of each activity.

How egocentric video is recorded

The camera sits close to the eyes so the footage matches what the wearer sees. The usual set-ups are these:

1. A head-mounted camera, such as a small action camera on a head strap or cap. Ego4D's project page lists seven different off-the-shelf head-mounted cameras used for its collection, from GoPro to camera glasses.
2. Smart glasses or camera glasses: frames with a built-in camera, sometimes with extra sensors.
3. Depth or multi-sensor rigs, which research teams use to add depth or motion data. HOI4D, for example, is 2.4 million RGB-D egocentric video frames over 4,000 sequences of people handling objects.

Some collections record long, continuous sessions while the wearer goes about the task as they normally would. Others use shorter task sequences or clips. EPIC-KITCHENS-100 captures long-term unscripted activity in 45 kitchen environments using head-mounted cameras.

Why AI research uses egocentric video

Egocentric video is training data: the examples a model learns from. Our guide to what AI training data is explains the idea in plain words. Researchers value the first-person view for three reasons:

• It shows hands and objects up close. The frame catches the grasp, the tool and the object at working distance.
• It keeps the steps in order. Continuous footage shows how a task unfolds from one step to the next.
• It gives a viewpoint near a robot's. A robot's own camera also looks down at its hands and the object in front of it.

The last reason is why robot-learning research uses it. R3M pre-trained a visual representation on Ego4D human video and reported more data-efficient learning of downstream robot manipulation tasks. That shows human video can help robots learn. It doesn't show that video alone can teach a robot to act. Our guide to how human video helps robots learn covers the evidence and its limits, and egocentric video compared with teleoperation data explains where robot demonstrations still come in.

Well-known egocentric datasets

The two Ego4D participant counts come from different sources: the Ego4D paper says 931 camera wearers and the project page says 923 unique participants. If you cite them, quote each with its source. Our comparison of the main public egocentric datasets covers licences and access, which differ from one dataset to the next.

Common mistakes and misconceptions

• People often equate POV video with vlogging. Creator POV clips are edited for viewers, but POV only describes the viewpoint. Research collections may use continuous sessions or task clips, with documentation and annotations suited to their research questions.
• Not every first-person clip works for training. What matters is what's in frame and whether the task runs start to finish. Our egocentric dataset spec checklist lists the fields to specify.
• Egocentric doesn't always mean first-person only. Some well-known datasets, such as Ego-Exo4D, pair first-person and external views.
• Bigger numbers don't always measure the same thing, because hours, frames and sequences are different units. HOI4D counts frames; Ego4D counts hours.
• Don't expect video alone to teach a robot to act. Human video can help, but the research still pairs it with robot data.

How TaskCapture records egocentric video

For physical work, our collectors wear a small camera on a head strap or cap, hands free. They record three data types: residential, commercial and gaming. Gaming is the screen plus every keyboard, mouse or controller input in sync, with no camera. Every identifiable person on camera signs a release, and commercial sites give written permission, as the Privacy Policy sets out. If you'd like to record your own work, the collector page explains how projects run.

Questions