Egocentric video vs teleoperation data: what each teaches a robot

Four terms, defined

Teleoperation data is recorded while a human operator controls a robot, either remotely or through a leader device the robot mirrors. Log the demonstration properly and the robot's camera frames line up with the commands it executed, so the data is already in that robot's action space.

Egocentric video is shot from a person's own point of view, usually with a head-mounted camera, while they do a task with their own hands. Our explainer on what egocentric (POV) video is covers how it's recorded and the best-known datasets.

The embodiment gap is the difference between the body in the data and the body of the robot that has to learn from it. A human hand, a two-finger gripper and a five-finger robot hand each reach, grasp and see the scene differently. Human video needs a mapping before a robot can act on it.

Action labels record, frame by frame, the action taken. In teleoperation they're the robot's joint or gripper commands. In gameplay recorded with inputs they're the keyboard, mouse or controller events. Plain egocentric video has none, so the actions have to be inferred from what the hands do.

What teleoperation data is, and what it is good for

For the robot that recorded it, teleoperation data is about as close to ground truth as you'll get. A person drives the actual hardware, so a logged demonstration shows what that robot saw and which commands it received. Nothing has to be translated from a human body to a robot. That makes it a natural fit for imitation learning, where a policy learns to copy demonstrations. ALOHA is the well-known example: a low-cost bimanual system that learns fine manipulation end to end from real demonstrations, collected through a custom teleoperation interface. Its authors call the algorithm Action Chunking with Transformers.

The price is coverage. Every hour needs a robot, an operator and a place to work. DROID shows what breadth costs: 76,000 demonstration trajectories across 564 scenes and 84 tasks, gathered by 50 data collectors in North America, Asia and Europe over 12 months. Its authors report that training on this more diverse data gave policies with higher performance and better generalization.

The other route is pooling data across labs. Open X-Embodiment assembled data from 22 different robots through a collaboration of 21 institutions, covering 527 skills. Pooling brings its own embodiment gap, because a demonstration from one robot isn't in another robot's action space. Our map of where robot training data comes from puts teleoperation next to the other sources, and our overview of imitation learning and VLA research covers the models trained on these datasets.

How human video enters robot learning

Egocentric video supplies the breadth teleoperation lacks. People already do the tasks robots are meant to learn, in kitchens, warehouses and workshops, and a head-mounted camera catches their natural hand-object interaction without a robot in the room. The Ego4D paper reports 3,670 hours of daily-life video from 931 camera wearers in 74 locations across 9 countries.

Researchers use that video in at least three ways. None of them removes the need for robot data.

• R3M used it for visual representations. It pre-trained on Ego4D and reported more data-efficient learning of downstream robot manipulation tasks than training from scratch. The robot still learns its actions from its own demonstrations, and the human video shapes how it sees.
• VIP turned Ego4D human video into rewards and progress signals. Without fine-tuning on task-specific data, it produced a frozen representation that gives a dense visual reward for simulated and real-robot tasks.
• MimicPlay went after high-level plans. It learned latent plans from human play data (video of people freely interacting with the environment with their hands) and used them to guide a low-level controller trained on a small number of teleoperated demonstrations.

In all three, human video can help by making the robot data go further. Our guide to how human video helps robots learn covers this research and its limits. For what each dataset contains and how it's licensed, see our comparison of the public egocentric datasets.

Teleoperation data vs egocentric video at a glance

Gameplay with inputs: video with the actions attached

Gameplay recorded with every keyboard, mouse or controller input is a third kind of data. The screen shows what happened, and the input log shows the action that caused it, frame by frame. Plain egocentric video has no such pairing.

Video PreTraining (VPT) shows why the pairing matters. Its authors used a small amount of labeled data to train an inverse dynamics model, then used that model to label a large source of online videos of people playing Minecraft. The agents they trained act through the native mouse and keyboard interface. That makes action-labelled gameplay useful for agents and models that have to learn what an action does on screen.

TaskCapture records gaming sessions this way: the game screen in sync with every keyboard, mouse or controller input, with no camera and no voice, chat, gamer handles or game telemetry.

When to use which

The two approaches complement each other. Match the data to the job:

Whether a public dataset already covers your setting is a separate question. Our guide to when public data is enough and when to commission custom collection helps you answer it.

Common mistakes

• Teams treat the two as substitutes. Human video carries no robot actions and teleoperation rarely reaches many sites, so plan for both if the robot has to work widely.
• It's easy to assume pooled robot data has no gap. A demonstration from one robot isn't in another robot's action space, even inside the same dataset.
• Buyers judge a dataset by its hours, which say nothing about framing, task coverage or whether the metadata matches the video. Run a checklist for judging a video dataset before you buy.
• Teams skip the licence. Public egocentric datasets carry their own terms, and some, such as EPIC-KITCHENS, are published for non-commercial use only.
• Footage from the wrong places is a quieter mistake: video from homes may not prepare a robot for a warehouse. Record or pick footage that looks like the place the robot will work.

Consent, and questions to ask any data vendor

Whoever you buy from, these five questions separate usable data from a liability:

1. Who recorded it, where, and under what agreement?
2. Did every identifiable person sign a release, and did every commercial site give written permission?
3. How is quality checked before delivery, and against what criteria?
4. What metadata comes with the video: device, timestamps, location, action logs?
5. What does the licence let you do with it, and is anyone barred from identifying the people in it?

Our own answers are short. Every identifiable person on camera signs a release, and every commercial site gives written permission. Footage reaches buyers only after quality review, and buyers are contractually barred from identifying anyone in it. The Privacy Policy sets out these rules in full. The custom egocentric data collection page covers data types, metadata and formats.

Questions