- Teleoperated demonstrations can support imitation learning from robot experience.teleoperation and imitation learning study
- Physics simulation can run alongside policy training to produce simulated experience.GPU physics simulation study
- Generative environment research explores action-controllable virtual worlds learned from video.generative environment study
- Human video can support visual representation learning; shared datasets can aggregate experience across robots.visual pretraining and cross-robot dataset studies
Five categories, without a supplier ranking
The egocentric pretraining comparison splits human observations from robot demonstrations. Hold onto that split when you read a data proposal. Ask what was recorded or generated, what supervision comes with it and which part of your learning system will use it.
The categories overlap. A teleoperated demonstration can end up in a public dataset later. A hub can host human video or robot trajectories. "Public" tells you about access. It isn't a sensor modality, and it promises nothing about permitted use.
What each category supplies
| Category | Input or experience | The question to resolve |
|---|---|---|
| Teleoperation fleets | Demonstrations produced by people controlling robots | Are the recorded observations and actions compatible with the intended policy? |
| Simulation and synthetic data | Experience or rendered observations from a modeled environment | What aspects of the real task must be checked separately? |
| Generative world models | Learned, action-controllable virtual environments | How will generated transitions be validated for the intended experiment? |
| Human activity video | Observations of people interacting with their surroundings | Does the method need visual features, progress signals or plans? |
| Public datasets and hubs | Existing collections distributed for reuse | What are the underlying data type, scope, release and terms? |
Put the same questions to every proposal. "Training data" on its own leaves too much open. A collection of human observations and a set of robot action trajectories can both be useful while doing different jobs.
Teleoperation and simulation: define the control evidence
A study of imitation learning from teleoperated demonstrations collects real robot demonstrations through a custom interface and learns sequences of actions. If you're buying, that points to a concrete spec: observation streams, action definitions, timing, robot configuration and the task being demonstrated. Collections won't all expose those fields the same way.
Physics simulation research runs simulation and policy training together on a GPU. A simulator gives the policy a modeled environment to act in. Whether you're buying or building, the question is which conditions you can model and which need separate evidence from the setting where you'll deploy.
Don't pick either one from an impressive sample video. Ask for the training input itself and how it goes into the method. The robot learning research guide has a reading checklist for telling a policy apart from its data and its evaluation.
Generated environments and human video: different learning roles
Generative environment research learns action-controllable virtual worlds from unlabelled video. It's one research example of a learned environment. It doesn't show that every generated interaction is good evidence for a physical robot task. Generated frames also shouldn't be counted as collected real-world experience.
For any proposed use, ask what the controls mean, what the generated sequence stands for and how you'll test it. Log the generation method and its inputs in the experiment record. An outcome that looks plausible still has to pass your evaluation before you accept it.
Research on human-video pretraining looks at another role: learning a visual representation that downstream robot policy learning can build on. The guide to how robots learn from human video explains why that helps even though it doesn't supply the whole controller. Settle on the role before you decide which activities to record.
Public collections and hubs: inspect what is underneath
A cross-robot dataset study combines experience from different robot platforms in standardized formats. It shows why documenting the component data and how it's organized pays off. Other public collections may still lack compatible actions, your task or the same permission to reuse the material.
Check the release and the original source terms, even when a hub makes finding and downloading easy. Our public egocentric dataset comparison shows how much activity scope, camera views and published access terms vary within one corner of the landscape.
In your own inventory, label two things: the underlying data type and the access route. "Existing human activity video, accessed through a research agreement" tells you far more than "open robot data." Leave open coverage or rights questions on the list until the right reviewer answers them.
Combine sources around an explicit experiment
- 01
State the target behavior
Name the task and the conditions in which you plan to evaluate it.
- 02
Assign a role to each source
Separate visual pretraining, robot action supervision and any simulated or generated experience.
- 03
Find remaining gaps
List missing activities, observations, control signals and documentation.
- 04
Choose a bounded addition
Compare custom and off-the-shelf data for that specific gap.
- 05
Evaluate the change
Keep a baseline and test whether the addition helped under the chosen conditions.
As an illustration, a plan might start from existing visual pretraining and then add demonstrations on the target robot. That's an experiment to run. It isn't a general recipe, and it doesn't predict an improvement. A buyer data request should say which input you want and what evidence would show it fits.
Common mistakes
- Comparing quantities in different units. Human video hours, robot trajectories and generated episodes don't add up.
- Treating the categories as mutually exclusive, when public distribution can apply to several underlying data types.
- Choosing a source before you name the learning stage and the signal that's missing.
- Taking a hub entry at face value. Trace the actual collection, release and permissions behind it.
- Assuming a hybrid plan will improve results. Evaluate the change and report where it helped and where it didn't.
Questions
Which category is best for robot training?
That depends on the task and the learning method. We give you questions for assessing inputs. We don't rank suppliers or call one category best across different experiments.
Are public datasets a separate kind of robot experience?
No. "Public" describes access. A public dataset can hold human video, robot demonstrations or other material, so find out its underlying data type and terms before you compare it with another source.
Can generated video replace real robot demonstrations?
The generative environment study cited here doesn't support that as a general claim. Spell out what generated experience would add, then test it against what the target task requires.
What should I ask a provider to send first?
Ask for the collection or generation method, sample files, field definitions, a coverage description, release information and the terms that apply. Also ask how they picked the sample and how the proposed delivery connects to your learning objective.