Embodied programming will transform robotics in the coming decade
Today I gave a talk at Brown’s Carney Institute for Brain Science called Embodied Programming Through Vision. Thank you to everyone who came, and to Carney for hosting me. This post is a written version of the talk.
Robots can learn how their own bodies move from vision alone, by watching themselves execute random commands. A video model can plan what should happen in the world, and a learned body program turns that plan into commands for one particular robot. The planner never needs to know any robot’s action coordinates. That is why I think embodiment generalization need not be a distinct learning problem.
Put a few motors on a desk lamp and you get a machine you can command. Is it a robot? My intuition is that it counts if we can control it to do different things. (I wrote more about this question in an earlier post.)


The question keeps coming up because robots no longer look like the rigid systems in textbooks. New manufacturing techniques enable new kinds of robots. The hand below has tendons and pressure-sensing fingertips, combines soft and rigid materials in one structure, and is 3D printed in one go by a single machine.
The pneumatic hand we worked with is also printed directly, from several materials and without assembly. It costs about 300 dollars to print


There was just one problem: we had no good way to control it. Controlling a robot means that, given a desired motion in space, such as pinching the index finger and the thumb, we find the commands that produce it. For this hand, that is how much air pressure to put in each tube.


Conventional robots avoid this problem by design. A Shadow Hand costs 150,000 to 200,000 dollars, and a UR5 arm over 30,000. They are expensive because conventional control needs precise sensors and motors in every joint, and very precise manufacturing. The controller infers the robot’s configuration from those joint sensors alone. It is like closing your eyes and guessing where your body is from muscle signals.


People, in contrast, are capable embodied programmers. We can learn to teleoperate a new robot within minutes, and the only sensors we need are our eyes. From vision alone, we build a model of the robot and predict how our commands will move it.
How have people tried to give robots this ability? Physics-based models need an expert to write down the geometry, the material properties, and simplifying assumptions, such as a backbone with constant curvature
Neural Jacobian Fields

The forward pass below makes this concrete: take a few points on the robot’s surface and a command, and the Jacobian field gives the 3D motion of those points. Each command moves its own part of the body. In the second figure, the field is predicted from one image of the pneumatic hand. Pick a command to see the motion it predicts.



Training needs no kinematic model and no demonstrations. We send random commands and film the robot with a few cameras. The model only watches how the robot looks and how it moves. After training, one camera is enough.
The training objective uses only those recordings. From a single image, the network reconstructs a Jacobian field and a radiance field. Given the command, the Jacobian field gives a 3D motion field. We volume-render both fields into a camera, and supervise the renders with the images and with the optical flow between two consecutive frames.









Why render at all? If we give the model depth as input and drop the rendering, its flow error is 7.5 times higher, and its Jacobians blend several actuators together. Our explanation is that without rendering into the other cameras, it gets no multi-view supervision.

And why predict a Jacobian, rather than the motion directly? With the same data and backbone, a model that predicts motion directly still fits the training data but falls apart on new commands. In simulation, its error on unseen commands is 26 to 57 times higher. On the real Allegro hand, its predicted motion smears across the whole hand.
We tested the same recipe on very different bodies: the 3D-printed pneumatic hand, the 16-DoF Allegro hand, a soft, compliant wrist made of handed shearing auxetics (HSA), and a low-cost 3D-printed arm. Each one learns its own Jacobian field from its own experience. For every body, the model recovers the 3D shape and the kinematic structure from vision alone.



With a Jacobian field, control becomes an optimization. We specify the motion we want, in 2D or 3D, and solve for the command whose predicted motion matches it. Gradient-based optimization finds that command in about half a second. The second figure below shows the optimization running on the pneumatic hand.


For longer motions, we close the loop. At every step, the robot looks at the current image, solves for a small command toward the next waypoint, executes it, and looks again.

The same recipe works on very different bodies. The pneumatic hand closes its fingers while part of it is hidden behind objects. The low-cost arm traces the letter M in the air. The soft wrist rotates while carrying a 350 g load: the load tilts it at rest, and the visual feedback still drives it to its goal, about 7 mm from the target.

Since each body learns its own model from its own experience, the hardware no longer has to be designed around our ability to model it by hand. Stepping back, the Jacobian field turns a motion plan into one body’s commands. I call this a body program. So far the motion plan came from us, because there was no planner. The second half of the talk was about learning that level too.



VERA


The planner is Wan, a pretrained 14-billion-parameter video diffusion transformer. We did not change its architecture. Camera views are tiled into one wide frame, and we post-train the model on robot videos from DROID, only to predict future frames. There is no action head. Actions come from the body program.


The instruction decides which plan the model generates. Pick one below: the observed frames stay the same, and only the words change.
The body program starts from VGGT
Why start from a pretrained video model? Internet video already teaches a lot about motion and interaction. When we trained the planner from scratch on the same robot videos, its held-out prediction error was 3 times higher (0.40 vs. 0.13). One way to picture this, borrowing from Neural Thickets


What does post-training keep? In our tests, the planner kept working with new camera viewpoints, new object arrangements, camera-angle changes, and lighting changes. These are examples, not a controlled comparison. My intuition is that we adapt behavior on top of a broad visual foundation, instead of learning each situation from scratch.




Does the Jacobian structure still matter once there is a planner? We compared three heads with the same backbone, model size, and data. The Jacobian head stays accurate as we add degrees of freedom, and it needs less data. In closed loop in simulation, it succeeds 70% of the time on the 16-DoF hand and 94% of the time on the 7-DoF arm, where a UniPi-style direct inverse dynamics model succeeds 0% of the time on both.
A few runs from our evaluations. These are examples, not controlled comparisons. First, place the yellow cube on top of the lego block. On the left is the generated plan, on the right what the real robot did.
The same planner also drives a 16-DoF hand. In simulation, only the prompt changes. Here the body program outputs 16 joint commands.
And on the real hand, with the clockwise prompt. The left two panels are the plan from the video model, and the right two show the real hand executing it.
Push the blue button. At the start, only one of the external cameras can see it, and the orange button is a distractor. The wrist camera first looks at the orange button, then finds the blue one and presses it.
Finally, we moved the external camera between the two settings below, and asked the robot to place the yellow cube on the white plate. Halfway through, someone moves the plate. In this run, the robot follows the plate and still places the cube on it.


Video models are becoming robot policies in industry too, for example Rhoda AI’s Direct Video-Action Model and NVIDIA’s GR00T 2 world action model. So the question is less whether to use a video model, and more where the shared model should meet the body.
The usual way to share one model across robots is a generative action model with one large action placeholder. RDT-1B defines a 128-dimensional action vector whose slots have fixed physical meanings: arm joints, grippers, end-effector pose, and the base. Each robot fills in the slots that match its actions, and the rest are padded. Our Allegro hand has 16 finger joints and our pneumatic hand has 15 pressure channels, and neither has a slot. π0 instead zero-pads every robot’s actions to the size of the largest robot, so the same output means different things on different robots. Either way, the shared model has to output each body’s own coordinates.
We do the opposite. The shared program never outputs an action. It outputs effects in the world, a video plan, and each body gets its own body program, learned from its own experience. Try each of our four bodies both ways.


This is the argument of the talk. Embodiment generalization need not be a distinct learning problem. Much of the problem comes from describing behavior in the coordinates of one particular body. If we describe behavior by its effects in the world, shared programs can act through different bodies, and each body can learn its own mapping from experience.
Everything above takes an instruction and moves a body. So the whole system can be one tool. A multimodal agent, such as Claude, could call act("put a plate at each seat") the way it calls any other tool, and break a long task into such calls. Programs have a recursive structure: a whole system today becomes one call inside a larger program.
act(instruction) 




act(...) call runs everything in this talk: a video plan, then a body program. We have not built this part yet. NVIDIA already describes an agent that controls GR00T 2 as a tool through an API, so I expect this to move quickly. This is where I think embodied programming is going in 2027.
Thank you to the Carney Institute for hosting me, and to everyone who came. Neural Jacobian Fields is joint work with Annan Zhang, Boyuan Chen, Hanna Matusik, Chao Liu, Daniela Rus, and Vincent Sitzmann. VERA is joint work with Evan Kim, Xingjian Bai, Tong Zhao, Tao Pang, Max Simchowitz, and Vincent Sitzmann. I am advised by Vincent Sitzmann and Josh Tenenbaum.
Neural Jacobian Fields: Paper · Project page · Code
VERA: Paper · Project page · Code
Please cite this post as:
Sizhe Lester Li. "Embodied Programming Through Vision." November 2026. https://sizhe-li.github.io/blog/2026/embodied-programming-through-vision/
or, in BibTeX:
@misc{li2026embodiedprogramming,
title = {Embodied Programming Through Vision},
author = {Li, Sizhe Lester},
year = {2026},
month = nov,
howpublished = {\url{https://sizhe-li.github.io/blog/2026/embodied-programming-through-vision/}},
note = {Blog post}
}
# goal: set the table for two