Chapter 9
The Five Data-Collection Strategies, Compared
7 sections · about 5 minutes
9.1 Real-robot teleoperation fleets
The gold standard, and the slowest. Perfectly matched action space, correct sensor statistics, real contact dynamics — and one human, one robot, one hour, in real time. DROID needed 50 collectors and twelve months for 350 hours; AgiBot needed 100 robots and a purpose-built building for ~3,000 hours.
Skild AI states the objection most bluntly: "Teleoperation happens in real-time. Even if we mobilized a global workforce to 'drive' robots 24/7, the time required to reach the trillions of tokens equivalent to an LLM is mathematically unfeasible" — and the data it produces is "trapped in sterile labs."
Who bets on it: TRI, AgiBot, Figure (roughly 500 hours for Helix), 1X, Physical Intelligence, UBTech, Scale AI's data factories.
9.2 Simulation and sim-to-real RL
Effectively free per hour and massively parallel. Figure's Helix 02 trained across 200,000+ parallel environments; Skild claims roughly 1,000 simulated years across 100,000 robot bodies.
It works spectacularly for locomotion and whole-body control. Boston Dynamics reports the sim-to-real gap is "very small" for Atlas, thanks to only two actuator types and a symmetric design. It works poorly for contact-rich manipulation, for the reason Goldberg identifies: submillimetre inaccuracies destroy contact predictions, and friction, deformables and compliance are all poorly modelled.
DrEureka (RSS 2024, UPenn/NVIDIA) automates the design of randomisation parameters using LLMs, which removes much of the hand-tuning that made domain randomisation an art.
Who bets on it: NVIDIA, Boston Dynamics, Skild, Galbot, Genesis AI, most locomotion groups.
9.3 Human egocentric video and wearables
Cheapest per hour by orders of magnitude, and unbounded in diversity. Ego4D (3,670 h), Ego-Exo4D (1,286 h), EgoDex (829 h), and corpora in the tens of thousands of hours.
The problem is fidelity: no forces, no torques, no robot-executable actions, and a visual and morphological gap. But the leverage is real. Meta's V-JEPA 2 shows 1M+ hours of video pretraining plus only 62 hours of robot data yielding 65–80% zero-shot pick-and-place on unseen objects. Generalist's GEN-1 makes the strongest claim: half a million hours of human wearable data with zero robot data in pretraining.
Who bets on it: Meta, Skild, Tesla, 1X, Generalist AI, NVIDIA (GR00T N1.7's 20K EgoScale hours).
9.4 Handheld and exoskeleton interfaces in robot-compatible action space
The clever middle, and in this book's judgement the most underrated of the five. The idea is to make the human's demonstration already be in the robot's action space, eliminating retargeting.
- UMI records 111 demos/hour versus 35 for SpaceMouse teleoperation — 3× faster, about 48% of bare-hand speed — with zero-shot transfer across UR5e and Franka.
- DexUMI extends this to multi-fingered hands with a wearable exoskeleton plus robot-hand video inpainting to close the visual gap, at 86% average success.
- DEXOP is the purest version: a passive exoskeleton mechanically coupling human fingers to robot fingers, giving true contact proprioception plus full-hand tactile capture, with significantly higher task performance per unit collection time than teleoperation.
- Sunday Robotics' Skill Capture glove is the commercial version, with the hand geometry matched to the robot's.
X Square Robot claims a UMI-style pipeline reaches all-robot-dataset performance at roughly 20× lower cost, with an 85% data-validity rate after physical replay validation.
The catch: the interface is hand-shaped, so it is embodiment-locked in a different way. Sunday's glove works because Sunday controls the hand.
9.5 Deployment flywheels and real-robot RL
Goldberg's preferred path, and the one with the best empirical support. Ambi Robotics accumulated 22 years of real robot data in four years while sorting 100M+ packages. Dexterity's Foresight was trained on 100M+ autonomous production actions. Chef Robotics has 100M meal servings. 1X's Redwood trains on autonomous episodes including failures.
Real-robot RL is now practical at small scale. Physical Intelligence's online RL work (March 2026) reports up to 3× speedups on contact-rich insertion from about fifteen minutes of real robot data per phase — roughly two hours wall-clock — with "half of the trials from the final RL policy... faster than any teleoperated demonstration." Berkeley's HIL-SERL does human-in-the-loop RL on motherboard assembly and timing-belt insertion.
The catch: deployment data is narrow by construction. Serve Robotics' 2,000-robot fleet produced navigation data that did not transfer to manipulation, which is why it had to buy Diligent.
9.6 Comparison table
| Strategy | Cost/hour | Throughput | Fidelity | Action-space gap | Diversity | Principal backers |
|---|---|---|---|---|---|---|
| Real teleop fleets | Highest | 1× realtime | Perfect | None | Low (labs) | TRI, Figure, 1X, PI, AgiBot |
| Simulation / sim2real | Near zero | Massively parallel | Good kinematics, poor contact | None (same robot) | High (synthetic) | NVIDIA, Skild, BD, Galbot, Genesis |
| Human egocentric video | Lowest | Unbounded | No forces, no actions | Severe | Highest | Meta, Generalist, Tesla, Skild |
| Handheld / exoskeleton | Low | ~3× teleop | Good; DEXOP adds tactile | Eliminated by construction | High (in the wild) | Song, Agrawal, Sunday, X Square |
| Deployment flywheel | Negative (revenue) | Fleet-scale, 24/7 | Perfect, includes failures | None | Narrow by task | Ambi, Dexterity, Chef, Dyna, RightHand |
(No reliable public dollar-per-hour figures exist for any of these; vendors do not publish them. The throughput ratios above are sourced, the cost column is ordinal.)
9.7 The strongest argument against everything except (a) and (e)
Sergey Levine's essay "Sporks of AGI" is the sharpest statement of the case for real robot data, and it is worth understanding precisely because Levine's own company would benefit commercially from the cheaper alternatives being true.
His argument is about intersections. Any surrogate domain — simulation, human video, an exoskeleton — overlaps with the real robot domain but does not coincide with it. A model trained on the surrogate fits both the shared structure and the discrepancies. And crucially: "as more powerful models fit the patterns in the data more tightly, they'll increasingly fit to the discrepancies." Scaling a surrogate-trained model makes the problem worse, not better.
Applied to human video specifically: a model trained on human data "will try to predict how a human will approach this problem, rather than predicting an effective strategy for a robot."
His conclusion: "real data is indispensable." Whether that is a law of nature or a temporary engineering fact is, arguably, the single most important open question in the field.