Imagine you wanted to build a machine that beats Magnus Carlsen at chess. You could collect every game Magnus ever played and train on that. You’d get a decent player, but you would not (generally) beat Magnus. And what if the goal is not just to beat Magnus, but to beat Magnus in 8 moves, every single time you play? AlphaGo started from human games, and then there was move 37, and now it’s better than any human. AlphaZero didn’t bother with human data, and it was stronger right away.
Somewhat bizarrely, parts of the humanoid industry are still running the Magnus plan. The pitch goes: humans are dexterous, there are billions of hours of video of humans doing things with their hands, so scale egocentric video, add a few thousand hours of teleoperation, and dexterity falls out.
Stating the obvious, at best this gets you to roughly human. It does not get you past human, which is where the business is. Nobody wants to pay $150k for a robot that folds laundry about as well as a tired person. The economic case for physical AI is a machine that is decisively faster, safer, and more precise than we are. Imitation learning by construction lands slightly below the demonstrator. What actually produces superhuman dexterity?
1/ What’s in a video of a human hand
A video is a record of where things were, at 30 frames per second, from one viewpoint, with occlusion every time the fingers wrap around something. Here’s what’s not in there:
No forces. The difference between a secure grasp and a dropped egg is a few newtons of normal force and the friction cone at each contact, and none of that shows up in pixels.
No contact geometry, meaning where the fingertip actually touches the object and how that patch deforms, which is the state variable that matters most for in-hand manipulation.
No failures, because people film themselves succeeding. The near-misses and recoveries, which are the most informative part of any dataset for a control problem, get edited out or never happen in the first place because the human corrected before the camera could tell.
And there’s a subtler problem. Human motion is optimal for human hardware, but human hardware is not particularly compelling. Tendon-driven fingers with ~25 degrees of freedom, muscle actuators with a roughly 50 ms latency, proprioception that’s good to maybe a degree, and a visual loop that runs at about 10 Hz for fine corrections. Your robot has none of those constraints and a different set of its own. Copying the human trajectory means importing all of the compensations the human made for their own hardware. It’s the Navier-Stokes point again: if you wanted a machine to fly, you would not start with video of people flapping their arms.
Where video does help is semantics. It tells the policy what "put the cup on the shelf" means, roughly what order the sub-steps go in, and what a reasonable approach vector looks like. That’s a useful prior, but it’s also the one thing LLM-style pretraining already does well, so the marginal value of the ten-millionth hour is effectively zero.
2/ Every superhuman dexterity result so far came from simulation
The manipulation demos that did something a human can’t, as opposed to the ones that merely look nice in a launch video, share a common origin.
The canonical one is OpenAI’s Dactyl in 2019: a Shadow hand solving a Rubik’s cube one-handed, under perturbation, with the policy trained entirely in simulation using automatic domain randomization. The finger gaits it discovered looked nothing like how a person turns a cube, which is what you’d expect from a policy that was never shown a person. Since then the pattern has repeated: in-hand reorientation of arbitrary objects from the Agrawal group at MIT, the Eureka work at Nvidia where an LLM writes the reward function and the policy spins a pen faster than most humans can, and essentially every quadruped locomotion controller that works on real terrain. All of them are sim-to-real RL with zero human motion capture anywhere in the pipeline.
Simulation wins over the real world because dexterity is a search problem over contact modes, and searching it takes millions of trials. A real robot gets you (optimistically) a thousand attempts a day and breaks things. Isaac Lab on one GPU is orders of magnitude better – a hundred thousand environments in parallel.
The main caveat of the current simulators is that they are only good at rigid bodies. MuJoCo, Isaac, and Genesis all handle a cube or a pen. They fail on cloth, cables, liquids, skin, food, and anything that deforms or tears. Teleoperation is a temporary band-aid but not the solution – the long term solution will be to add the physics into the simulators. Whoever closes the deformable-contact sim gap gets the next step change in dexterity, and that’s a physics problem.
3/ Sense what humans can’t
Human fingertips have roughly 100 mechanoreceptors per square centimetre at the tip and the fast-adapting ones top out somewhere around a few hundred hertz. Our conscious correction loop for fine manipulation runs at roughly 10 Hz. That’s the very marginal biological sensing system your training data was generated by.
A GelSight-style optical tactile sensor gives you a full contact-patch image with sub-millimetre geometry: you can read the texture of the object, the shear field, incipient slip before the object actually moves. Capacitive and MEMS arrays close loops at a kilohertz. Six-axis force-torque at the wrist is a solved product. A robot with these sensors has super-human state information, so no human demonstration encodes how to use it. Birds have no use for jet turbines.
The policy has to be learned, and the only two ways to learn a tactile policy are simulation (see section 2, above; tactile sim is even less mature than rigid-body contact) or the robot’s own experience. There is no YouTube of tactile data. This is the second problem with the scale-video argument: video is the modality that’s cheapest to collect and least informative about contact, so scaling it optimizes the wrong thing. Yes sure, searching for your keys at night next to the lamppost is certainly easier than getting a flashlight.
We see a small version of this every week with our own fleet. The quadrupeds carry thermal, LiDAR, and other sensors that a human patrol officer or solider doesn’t have. Nobody trains the patrol policy by watching security guards walk around. You train it on what the sensors see, in the environment, with the robot’s own kinematics. Manipulation is the same story with harder physics.
4/ Practice beats watching
The most interesting result of the past year was Physical Intelligence’s π*0.6 and RECAP in November 2025 (paper). They took a π0.5-class policy trained on demonstrations, let it run autonomously, trained a value function on the robot’s own successes and failures, and used advantage conditioning to push the policy toward the actions that actually worked. On the hard tasks, laundry in real homes, box assembly, pulling espresso shots, throughput more than doubled and failure rates halved. The hardware and the demonstrations were unchanged; the robot’s own experience was the only new input.
The PI team is explicit about the mechanism: imitation suffers from compounding error. One small mistake puts the robot in a state slightly outside the demonstration distribution, where it’s likely to make a bigger mistake, and then your latte gets tipped over in the CafeX delivery hatch at SFO. More demonstrations of the clean path don’t help with this. What helps is experience of the messy path plus a signal saying which recovery was better, and that signal only exists if the robot is allowed to fail.
This is the robotics version of what happened to language models. Pretraining on human text gets you to a strong imitator. RLHF and then RL on verifiable rewards is what took the models past the median human on coding and math. Nobody argues that the way to get a model that beats humans at competition math is to scrape more human solutions. The analogous move in robotics is a VLA prior plus on-robot RL, and the data comes from the fleet rather than from a person wearing a GoPro.
It also happens to be the commercially sensible path, because every deployed robot is already producing this data. A fleet of 100 robots running eight-hour shifts generates 800 robot-hours of experience a day, in the actual environments where the policy has to work, with real failures and real recoveries.
5/ A requiem for five-fingered hands – three fingers are better
Aside from human vanity, the five-fingered humanoid hand exists so the robot can be trained on, and compared to, human hands. The shape is dictated by a legacy data pipeline. Tesla, Figure, and the Chinese humanoid makers built anthropomorphic hands because their training strategy (at least in 2023/24) required a retargeting map from human video, and that map only works if the kinematics roughly match.
When you drop the human-video requirement, then you can actually start to build more useful manipulators. A Shadow hand has 20-odd actuated degrees of freedom, dozens of tendons, and a maintenance schedule. A three-fingered hand with direct-drive or quasi-direct-drive joints, like Shadow’s own DEX-EE or the grippers the warehouse companies converged on, has an order of magnitude fewer parts, is far easier to simulate accurately (fewer contacts, no tendon slack), and can run force control at 1 kHz or above because there’s no compliance hiding between the motor and the fingertip. Speed in particular is almost entirely a hardware question. Control bandwidth, actuator torque density, and the speed of sound in your linkage set the ceiling, not some random outcome of biological evolution.
The fastest path to superhuman dexterity is to design the hand and the simulator together, then let RL discover gaits that have no human analogue. The Dactyl cube-turning gait was a preview. Expect strategies that use the back of the fingers and the table as a third hand, and run at speeds a human observer can’t follow. It won’t (and should not) look like us, for the same reason an A320 doesn’t flap.
What’s the recipe?
-
Human data supplies an initial, generally poor, semantic prior.
-
A physics engine that handles contact correctly, including soft contact.
-
Super-human tactile and force sensing.
-
On-robot RL, so the fleet’s real-world failures become the training set.
-
Hardware designed around the above rather than around what a camera can retarget from egocentric data.
At OpenMind we’ve written down our definition of the physical-AI watershed moment before: a robot that, given only the laws of physics and knowledge of its own body, learns to move and manipulate within 24 hours of walking out of the lab. Over-reliance on egocentric data is a barrier to that.
