Categories
Uncategorized

RIP Egocentric video — it’s time to move on and focus on superhuman dexterity

Imagine you wanted to build a machine that beats Magnus Carlsen at chess. You could collect every game Magnus ever played and train on that. You’d get a decent player, but you would not (generally) beat Magnus. And what if the goal is not just to beat Magnus, but to beat Magnus in 8 moves, every single time you play? AlphaGo started from human games, and then there was move 37, and now it’s better than any human. AlphaZero didn’t bother with human data, and it was stronger right away.

Somewhat bizarrely, parts of the humanoid industry are still running the Magnus plan. The pitch goes: humans are dexterous, there are billions of hours of video of humans doing things with their hands, so scale egocentric video, add a few thousand hours of teleoperation, and dexterity falls out.

Stating the obvious, at best this gets you to roughly human. It does not get you past human, which is where the business is. Nobody wants to pay $150k for a robot that folds laundry about as well as a tired person. The economic case for physical AI is a machine that is decisively faster, safer, and more precise than we are. Imitation learning by construction lands slightly below the demonstrator. What actually produces superhuman dexterity?

1/ What’s in a video of a human hand

A video is a record of where things were, at 30 frames per second, from one viewpoint, with occlusion every time the fingers wrap around something. Here’s what’s not in there:

No forces. The difference between a secure grasp and a dropped egg is a few newtons of normal force and the friction cone at each contact, and none of that shows up in pixels.

No contact geometry, meaning where the fingertip actually touches the object and how that patch deforms, which is the state variable that matters most for in-hand manipulation.

No failures, because people film themselves succeeding. The near-misses and recoveries, which are the most informative part of any dataset for a control problem, get edited out or never happen in the first place because the human corrected before the camera could tell.

And there’s a subtler problem. Human motion is optimal for human hardware, but human hardware is not particularly compelling. Tendon-driven fingers with ~25 degrees of freedom, muscle actuators with a roughly 50 ms latency, proprioception that’s good to maybe a degree, and a visual loop that runs at about 10 Hz for fine corrections. Your robot has none of those constraints and a different set of its own. Copying the human trajectory means importing all of the compensations the human made for their own hardware. It’s the Navier-Stokes point again: if you wanted a machine to fly, you would not start with video of people flapping their arms.

Where video does help is semantics. It tells the policy what "put the cup on the shelf" means, roughly what order the sub-steps go in, and what a reasonable approach vector looks like. That’s a useful prior, but it’s also the one thing LLM-style pretraining already does well, so the marginal value of the ten-millionth hour is effectively zero.

2/ Every superhuman dexterity result so far came from simulation

The manipulation demos that did something a human can’t, as opposed to the ones that merely look nice in a launch video, share a common origin.

The canonical one is OpenAI’s Dactyl in 2019: a Shadow hand solving a Rubik’s cube one-handed, under perturbation, with the policy trained entirely in simulation using automatic domain randomization. The finger gaits it discovered looked nothing like how a person turns a cube, which is what you’d expect from a policy that was never shown a person. Since then the pattern has repeated: in-hand reorientation of arbitrary objects from the Agrawal group at MIT, the Eureka work at Nvidia where an LLM writes the reward function and the policy spins a pen faster than most humans can, and essentially every quadruped locomotion controller that works on real terrain. All of them are sim-to-real RL with zero human motion capture anywhere in the pipeline.

Simulation wins over the real world because dexterity is a search problem over contact modes, and searching it takes millions of trials. A real robot gets you (optimistically) a thousand attempts a day and breaks things. Isaac Lab on one GPU is orders of magnitude better – a hundred thousand environments in parallel.

The main caveat of the current simulators is that they are only good at rigid bodies. MuJoCo, Isaac, and Genesis all handle a cube or a pen. They fail on cloth, cables, liquids, skin, food, and anything that deforms or tears. Teleoperation is a temporary band-aid but not the solution – the long term solution will be to add the physics into the simulators. Whoever closes the deformable-contact sim gap gets the next step change in dexterity, and that’s a physics problem.

3/ Sense what humans can’t

Human fingertips have roughly 100 mechanoreceptors per square centimetre at the tip and the fast-adapting ones top out somewhere around a few hundred hertz. Our conscious correction loop for fine manipulation runs at roughly 10 Hz. That’s the very marginal biological sensing system your training data was generated by.

A GelSight-style optical tactile sensor gives you a full contact-patch image with sub-millimetre geometry: you can read the texture of the object, the shear field, incipient slip before the object actually moves. Capacitive and MEMS arrays close loops at a kilohertz. Six-axis force-torque at the wrist is a solved product. A robot with these sensors has super-human state information, so no human demonstration encodes how to use it. Birds have no use for jet turbines.

The policy has to be learned, and the only two ways to learn a tactile policy are simulation (see section 2, above; tactile sim is even less mature than rigid-body contact) or the robot’s own experience. There is no YouTube of tactile data. This is the second problem with the scale-video argument: video is the modality that’s cheapest to collect and least informative about contact, so scaling it optimizes the wrong thing. Yes sure, searching for your keys at night next to the lamppost is certainly easier than getting a flashlight.

We see a small version of this every week with our own fleet. The quadrupeds carry thermal, LiDAR, and other sensors that a human patrol officer or solider doesn’t have. Nobody trains the patrol policy by watching security guards walk around. You train it on what the sensors see, in the environment, with the robot’s own kinematics. Manipulation is the same story with harder physics.

4/ Practice beats watching

The most interesting result of the past year was Physical Intelligence’s π*0.6 and RECAP in November 2025 (paper). They took a π0.5-class policy trained on demonstrations, let it run autonomously, trained a value function on the robot’s own successes and failures, and used advantage conditioning to push the policy toward the actions that actually worked. On the hard tasks, laundry in real homes, box assembly, pulling espresso shots, throughput more than doubled and failure rates halved. The hardware and the demonstrations were unchanged; the robot’s own experience was the only new input.

The PI team is explicit about the mechanism: imitation suffers from compounding error. One small mistake puts the robot in a state slightly outside the demonstration distribution, where it’s likely to make a bigger mistake, and then your latte gets tipped over in the CafeX delivery hatch at SFO. More demonstrations of the clean path don’t help with this. What helps is experience of the messy path plus a signal saying which recovery was better, and that signal only exists if the robot is allowed to fail.

This is the robotics version of what happened to language models. Pretraining on human text gets you to a strong imitator. RLHF and then RL on verifiable rewards is what took the models past the median human on coding and math. Nobody argues that the way to get a model that beats humans at competition math is to scrape more human solutions. The analogous move in robotics is a VLA prior plus on-robot RL, and the data comes from the fleet rather than from a person wearing a GoPro.

It also happens to be the commercially sensible path, because every deployed robot is already producing this data. A fleet of 100 robots running eight-hour shifts generates 800 robot-hours of experience a day, in the actual environments where the policy has to work, with real failures and real recoveries.

5/ A requiem for five-fingered hands – three fingers are better

Aside from human vanity, the five-fingered humanoid hand exists so the robot can be trained on, and compared to, human hands. The shape is dictated by a legacy data pipeline. Tesla, Figure, and the Chinese humanoid makers built anthropomorphic hands because their training strategy (at least in 2023/24) required a retargeting map from human video, and that map only works if the kinematics roughly match.

When you drop the human-video requirement, then you can actually start to build more useful manipulators. A Shadow hand has 20-odd actuated degrees of freedom, dozens of tendons, and a maintenance schedule. A three-fingered hand with direct-drive or quasi-direct-drive joints, like Shadow’s own DEX-EE or the grippers the warehouse companies converged on, has an order of magnitude fewer parts, is far easier to simulate accurately (fewer contacts, no tendon slack), and can run force control at 1 kHz or above because there’s no compliance hiding between the motor and the fingertip. Speed in particular is almost entirely a hardware question. Control bandwidth, actuator torque density, and the speed of sound in your linkage set the ceiling, not some random outcome of biological evolution.

The fastest path to superhuman dexterity is to design the hand and the simulator together, then let RL discover gaits that have no human analogue. The Dactyl cube-turning gait was a preview. Expect strategies that use the back of the fingers and the table as a third hand, and run at speeds a human observer can’t follow. It won’t (and should not) look like us, for the same reason an A320 doesn’t flap.

What’s the recipe?

  1. Human data supplies an initial, generally poor, semantic prior.

  2. A physics engine that handles contact correctly, including soft contact.

  3. Super-human tactile and force sensing.

  4. On-robot RL, so the fleet’s real-world failures become the training set.

  5. Hardware designed around the above rather than around what a camera can retarget from egocentric data.

At OpenMind we’ve written down our definition of the physical-AI watershed moment before: a robot that, given only the laws of physics and knowledge of its own body, learns to move and manipulate within 24 hours of walking out of the lab. Over-reliance on egocentric data is a barrier to that.

Categories
Uncategorized

Architectural Gaps in Physical AI: When’s the Big Moment?

In some specific robotics use cases, the magic moment has clearly already happened. Autonomous driving had its moment in 2025. That’s why in 2026 you can ride a wheeled robot, a Waymo, all over SF with minimum hassle.

In Ukraine, autonomous drones now rule the sky (and the battlefield). At Amazon, more than 1 million robots help humans to move, package, sort, and pick, and we take it for granted that the stuff we ordered earlier today shows up at our doorstep the next morning. In China, humanoids now outrun Usain Bolt, the fastest-sprinting human.

But for many other use cases, it seems robots aren’t quite there yet. Is this just a question of needing to patiently wait until robots outperform humans, task by task, or are there deeper structural gaps? Based on growing real world deployments of our robots in SF and NYC, we see two gaps over and over again.

1/ We are using the wrong brain architecture

Engineering is about making choices. Remember the MIT saying – sleep, friends, grades – choose two? The same applies to robots. If you operate in the real world, you have a finite energy, constrained mass, and finite ability to dissipate heat. And if you want people to buy your robots, they need to be reasonably priced (unlike, say, anything ever manufactured by Boston Dynamics).

What this means for robot brains is that trying to build one all-capable model, the one to rule them all, is a fool’s errand. Sure you can prepend “universal”, “all powerful”, or “grand unified” to your approach, but that’s just a labeling exercise. The solution is to operate a smart router (with context) as the core of the brain, and then dynamically (re)optimize which data go to which models, as a function of the robot’s task and environment.

Need to balance on your toe? Focus all your resources on that stability task. Listening to a lonely elderly parent in the park? Well then use all your compute for empathetic personalized conversation and gesture generation. Helping a lost child find their parents at SFO? Now you care about path planning and scanning human (parent) faces for expression of concern.

As a capable robot, each of those tasks deserves your focus, and that’s why, as a robot, you should be constantly adjusting which models (and compute and actuators and sensors) you should be using in this very instant for optimal performance. At OpenMind, we are placing our bet on intelligent routing/attention as the foundational brain function of a robot, the goal being to use the best task-optimized model for the immediate task, whatever it may be.

2/ We left out the physics

As the name implies, physical AI (and robotics) is about physics, not just bits. Let’s say you want a machine to perform a specific task in the physical world, such as fly. Presumably, you would not start by collecting videos of humans flapping their arms. A more promising strategy is to focus on what actually constrains and enables that capability – Navier-Stokes allows you to compute what an efficient wing looks like. For human readers, don’t take this (too) personally, but the way in which you drive cars, fold t-shirts, do patient intake in the hospital ED, or pick up apples, may not be generally optimal given the underlying physics. The relevant question is – given a specific desired physical task/action, and given your current physical form (One arm? Two wings? Three feet? Two wheels?), and your current battery level, what is the optimal sequence of movements to accomplish that task? That’s a nonlinear optimization problem, not a “replay memorized motion sequences” task. The framing as nonlinear optimization gracefully accommodates changes in your current physical form and capabilities. If one of your arms falls off, no problem – you just change how you move, without more than a moment’s recalculation.

OpenMind’s definition of the Physical Turing Test

The physical Turing test (by Nvidia’s Jim Fan) builds on Alan Turing’s test and refers to the ability of a robot to operate in the real world without it being apparent to observing humans that they are seeing a robot rather than one of their own. This is one good way to define a discrete watershed, but it’s still very much centered on our self-reflection.

At OpenMind, we have a more general definition with three basic ingredients. We think that the “ChatGPT” moment for physical AI will be when robots have these three attributes:

1/ [Rapid learning of extreme dexterity] Given only the laws of physics and knowledge of its internal architecture and force generating capabilities, a robot must learn to move, run, jump, and manipulate diverse physical objects within 24 hours of being allowed to freely explore the real world. Further, the robot will gracefully and quickly overcome loss of one major sensor (e.g. a camera) or actuator (e.g. a finger, wheel, or foot) without external help.

2/ [Success with diverse cognitive, planning, and physical tasks] The robot will complete a succession of diverse tasks, encompassing verbal, emotional, reasoning, coordination, spatial navigation, manual/physical, and learning, without prior knowledge of which specific tasks it will face, and in which order.

3/ [Safety/Constitutional Robotics]. By virtue of its core design, the robot will reconcile the above capabilities with an internal constitution that minimizes risks of the technology for humans. You might well wonder how this can be possible – TLDR it’s probably not possible with our current approach to AI since we have very limited understanding of how frontier reasoning models actually work. It’s way too easy to hide behaviors inside of these models and they almost certainly will try to solve problems in ways that harm humans.