- Datasheets for Datasets proposes documenting motivation, composition, collection and recommended uses.Datasheets for Datasets
- Data Cards describes structured documentation covering dataset origins, methods and intended use.Data Cards
- EPIC-KITCHENS-100 includes evaluation of generalization to footage collected two years later.Rescaling Egocentric Vision
- A licensing audit of text datasets found missing and miscategorized licence information. It did not survey video quality.Data Provenance Initiative
Read the documentation before watching the highlight reel
The egocentric pretraining comparison starts from intended use, because visual pretraining and learning robot actions need different evidence. Write your use at the top of the review. Then ask the supplier to describe the specific release you would receive.
Datasheets for Datasets gives you a workable structure: why the dataset exists, what's in it, how it was collected and which uses its creators recommend. Data Cards extends the documentation discussion to stakeholders across the dataset lifecycle. Neither paper certifies that a particular dataset suits your model.
Turn that structure into a document request: release identifier, collection scope, file inventory, annotation definitions, known limitations and permitted uses. The dataset specification guide turns those questions into fields a supplier can fill in. Get the missing details before you let a polished sample stand in for the whole collection.
A practical quality checklist
| Check | How to inspect it | A reason to investigate further |
|---|---|---|
| Task visibility | Watch complete examples, including setup and outcome | The decisive interaction is outside the frame |
| Image usability | Inspect originals through your intended decoding pipeline | Samples differ from delivered files |
| Timing | Compare timestamps and any paired streams | Alignment is undocumented or inconsistent |
| Annotation meaning | Read definitions and inspect boundaries on examples | Reviewers interpret the same label differently |
| Coverage | Group samples by task, setting and recording session | A claimed category has no inspectable examples |
| Duplicate handling | Ask how clips and sessions are related | Near-identical material appears across evaluation splits |
| Provenance | Trace files to the documented collection and terms | The supplier cannot explain the source or permissions |
The table is a review method. It sets no universal pass mark, so pick thresholds for your own task. Decide what detail an annotator has to tell apart before you set a resolution requirement. A resolution label on its own doesn't prove the interaction is visible.
Check coverage and distribution separately from file quality
Build a coverage sheet with the activities, objects and settings your evaluation needs, and mark each cell present, absent or unknown. Leave "unknown" on the sheet where everyone can see it; it doesn't count as coverage. Ask whether the sample was picked to show variety or drawn to match the proposed delivery.
The EPIC-KITCHENS-100 paper treats time as an evaluation axis: it tests generalization from earlier recordings to footage captured two years later. That split is part of one dataset's research design. It isn't a rule every buyer has to follow when splitting video.
In your own experiment, decide whether to hold out people, sessions, locations or dates, and write the choice down before you look at model results. A dataset can decode perfectly and still miss an evaluation condition you care about. When the gap is specific, weigh custom and off-the-shelf data against that gap. Don't assume more footage of the same activity will close it.
Verify provenance, licences and consent evidence
The Data Provenance Initiative audited text datasets and reported omissions and errors in the licence information on hosting sites. Its figures don't measure video datasets, so don't quote them as if they did. The lesson for a buyer is narrower: read the source terms behind a catalogue label before you rely on the label.
On the video side, EPIC-KITCHENS' published terms are non-commercial. Being able to download it doesn't tell you whether it fits a commercial training plan. The licensing and consent checklist helps you put the evidence in order for a qualified reviewer. It's a documentation workflow and reaches no legal conclusion about your project.
Keep rights apart from visual inspection. A clip can be sharp and well labelled while its permitted use is still open. Log that as an open decision. If you average it into an overall quality score, nobody will see it.
Run a sample review you can repeat
- 01
Freeze the proposal
Record the release, intended use, task coverage and file formats under review.
- 02
Agree on sampling
Include ordinary examples and difficult cases, with the selection method recorded.
- 03
Inspect complete tasks
Note missing setup, hidden interactions, ambiguous endings and any private material requiring review.
- 04
Test the pipeline
Decode the actual files, load metadata and inspect the alignment of any paired streams.
- 05
Review labels
Have reviewers apply the written definitions and discuss disagreements on the same examples.
- 06
Make a decision
Separate acceptable material, remediable defects and unanswered questions before accepting a delivery.
Save an example of each issue with its file identifier and timestamps. A revision request then points at something specific: "the hand-object contact is outside the frame during this interval" is a statement a reviewer can check. On the collector side, the guide to what gets footage accepted covers the same concerns from behind the camera.
Common mistakes
- Watching only the supplier's highlights, without asking how that sample relates to the proposed delivery.
- Rolling every concern into one score. Image usability, coverage, labels and rights are separate decisions.
- Setting technical thresholds before you know the task. Decide first what has to be visible or synchronized.
- Treating good documentation as model validation. You still have to run the evaluation that motivated the purchase.
- Comparing offers from incompatible categories. The robot data landscape shows what each kind of offer supplies; put the inputs you need in a video data request.
Questions
What is the best measure of video dataset quality?
We don't propose a single one. Start with whether the data suits a defined use, then record file usability, coverage, annotations and permissions separately. The decision should show which requirements were met and which are still uncertain.
Should every dataset use the same resolution and frame rate?
No. The checklist sets no universal threshold. Work out the visible detail and timing your method needs, test real samples, and write the resulting requirements into the delivery specification.
How much sample footage should I review?
Enough to cover the conditions you plan to buy and the issues you need to catch. Write down what the sample can't tell you. A small sample can expose a defect without showing how often it occurs across the collection.
Does a public dataset page establish commercial rights?
No. Read the dataset's own terms. EPIC-KITCHENS is publicly available, and its published licence is non-commercial. Get your intended use reviewed before you lean on a general label like "public".