Custom vs off-the-shelf training data: how to choose

What off-the-shelf datasets give you

Off-the-shelf data already exists. Research consortia built the public egocentric datasets with documented protocols, annotations and benchmarks, so you can start training and compare your numbers with published work. Our egocentric video data page puts these public datasets and custom collection side by side in one table.

The draw is scale and breadth. The Ego4D paper describes 3,670 hours across hundreds of scenarios (household, outdoor, workplace, leisure) from 931 camera wearers. EPIC-KITCHENS-100 has 100 hours of unscripted kitchen activity in 45 environments, with about 90,000 annotated actions. For pre-training, baselines and benchmarks, that is hard to match. Our comparison of the main public egocentric datasets lists what each one contains.

Their limits

An off-the-shelf release is fixed, and three limits come up most often.

• Scope is whatever was recorded: those tasks, those places, those people. A particular workflow your model must handle, such as a restaurant prep line or a warehouse pick station, may show up for a few seconds or not at all.
• Licence terms vary by dataset. EPIC-KITCHENS says "You may not use the material for commercial purposes" under its public licence, though it offers a contact for commercial licences. The Ego4D licence agreement permits commercial or non-commercial product development within its purpose, but "without the right to sublicense". Our licensing and consent checklist lists what to verify.
• Format comes with the release. Resolution, frame rate, metadata and annotations are whatever was published, and your pipeline has to adapt to them.

Public data can still be the right choice. These limits tell you whether it is enough on its own.

Distribution shift: when your setting is not in the data

Train a model on one distribution, deploy it on another, and it can lose accuracy. The WILDS benchmark says distribution shifts, where the training distribution differs from the test distribution, "can substantially degrade the accuracy" of machine-learning systems deployed in the wild. WILDS is about machine learning in general and doesn't test robot video. The same logic holds for any setting your training data doesn't show.

Egocentric video has its own version of the problem. EPIC-KITCHENS-100 built a challenge around it: do models trained on 2018 footage generalise to footage recorded two years later? Run the same kind of test before you buy anything. Evaluate a model trained on public data on footage that looks like your deployment, and measure how far it drops.

When custom collection is worth it

Custom collection is not automatically better. It covers less and takes longer to start, and it only pays off when the spec targets the gap you measured. Write that spec with our egocentric dataset spec checklist.

A hybrid approach

You can use both. A common pattern:

1. Start with public data for pre-training and baselines, within its licence.
2. Measure the gap by evaluating on footage from your target setting.
3. Commission custom footage for that gap: the tasks, sites or regions the public data misses.
4. Keep the sources separate in your records, each with its own licence and documentation.

The same reasoning works across data types. Our guide to egocentric video compared with teleoperation data shows how teams combine broad human video with narrow robot demonstrations for robot learning.

Common mistakes

• Teams buy custom data before testing public data. Measure the gap first; you may need less than you think.
• A public download gets treated as a commercial licence. Read the terms, and check whether a separate commercial option exists.
• The spec is vague. "More kitchen video" repeats what public data already has, so name the missing tasks and places.
• Datasets get judged by hours alone. Run a checklist for judging a video dataset on a sample instead.
• Other sources get ignored. Simulation, robot fleets and teleoperation also supply training data, and our map of where robot training data comes from lays them out.

How TaskCapture collects custom data

We collect to order: you send a request, we match collectors who already do those tasks, they record the work, and you receive the footage after quality review. We offer three data types: residential, commercial and gaming with synchronized inputs. Video and metadata come in the format you ask for, and resolution, frame rate, minimum volumes and lead times are agreed per request. We confirm coverage when we scope your request.

Questions