Embodied Programming Through Vision

Embodied programming will transform robotics in the coming decade

Today I gave a talk at Brown’s Carney Institute for Brain Science called Embodied Programming Through Vision. Thank you to everyone who came, and to Carney for hosting me. This post is a written version of the talk.

TL;DR

Robots can learn how their own bodies move from vision alone, by watching themselves execute random commands. A video model can plan what should happen in the world, and a learned body program turns that plan into commands for one particular robot. The planner never needs to know any robot’s action coordinates. That is why I think embodiment generalization need not be a distinct learning problem.

Is this a robot?

Put a few motors on a desk lamp and you get a machine you can command. Is it a robot? My intuition is that it counts if we can control it to do different things. (I wrote more about this question in an earlier post.)

The question keeps coming up because robots no longer look like the rigid systems in textbooks. New manufacturing techniques enable new kinds of robots. The hand below has tendons and pressure-sensing fingertips, combines soft and rigid materials in one structure, and is 3D printed in one go by a single machine.

The pneumatic hand we worked with is also printed directly, from several materials and without assembly. It costs about 300 dollars to print, and once it is printed, we can mount it on a robot arm.

There was just one problem: we had no good way to control it. Controlling a robot means that, given a desired motion in space, such as pinching the index finger and the thumb, we find the commands that produce it. For this hand, that is how much air pressure to put in each tube.

What control means here
The pneumatic hand on a robot arm, then the desired motion drawn as arrows on the fingers, then the air tubes that carry the commands
  1. The 3D-printed pneumatic hand, mounted on a robot arm.
  2. A desired motion in space: pinch the index finger and the thumb (the arrows).
  3. The commands that produce it: how much air pressure goes into each tube.
Desired motions (arrows) and the commands that produce them (the air tubes).

Conventional robots avoid this problem by design. A Shadow Hand costs 150,000 to 200,000 dollars, and a UR5 arm over 30,000. They are expensive because conventional control needs precise sensors and motors in every joint, and very precise manufacturing. The controller infers the robot’s configuration from those joint sensors alone. It is like closing your eyes and guessing where your body is from muscle signals.

Conventional robots, click by click
A Shadow Hand and a UR5 arm with their prices, the inside of a UR5 actuator, and a rigid arm with a sensor at each joint
  1. A Shadow Hand and a UR5 arm, next to the printed hand.
  2. Inside a UR5 joint: a precise motor and sensor.
  3. Rigid links, with a sensor at every joint (A to F).
Conventional robots are designed to make their modeling and control easy.

People, in contrast, are capable embodied programmers. We can learn to teleoperate a new robot within minutes, and the only sensors we need are our eyes. From vision alone, we build a model of the robot and predict how our commands will move it.

Learning a body from vision

How have people tried to give robots this ability? Physics-based models need an expert to write down the geometry, the material properties, and simplifying assumptions, such as a backbone with constant curvature. Sensors and markers need the body to be instrumented, with sensors built into it or markers tracked by motion capture. Visual self-models learn the robot’s shape from cameras, but they take measured joint angles as input, so they still assume precise encoders. The printed hand has no joint sensors, and its soft materials are hard to model by hand. We wanted something that needs vision alone.

What each approach needs
A backbone model of a soft arm, and a soft arm holding an apple
Physics-based models
Della Santina, Duriez & Rus 2023
Needs
Expert knowledge
Geometry, material properties, and simplifying assumptions
A soft arm with embedded sensors
Sensors and markers
Truby, Della Santina & Rus 2020
Needs
Instrumentation
Embedded sensors, or markers tracked by motion capture
A robot arm imagining its own shape for different joint angles
Learned shape models
Chen et al., Science Robotics 2022
Needs
Precise sensors
Precise encoders, plus motion capture for fine control
The pneumatic hand on a robot arm, seen by one camera
Neural Jacobian Fields
Li et al., Nature 2025
Needs
Vision alone
Video of random commands to train; one camera to control
  1. Physics-based models: an expert builds the model by hand.
  2. Sensors and markers: the body itself is instrumented.
  3. Visual self-models learn the shape from cameras, but still take measured joint angles as input.
  4. Neural Jacobian Fields need vision alone. Nothing to model, build in, or measure.
What each approach needs before it can control a new body.

Neural Jacobian Fields need only vision: one image in, 3D motion out. From a single camera image, a network predicts a Jacobian field. For every 3D point, the field is a linear map from a small change in the command to the motion of that point:

\[\delta \mathbf{x} = \mathbf{J}(\mathbf{x}, I)\,\delta \mathbf{u}.\]

The forward pass below makes this concrete: take a few points on the robot’s surface and a command, and the Jacobian field gives the 3D motion of those points. Each command moves its own part of the body. In the second figure, the field is predicted from one image of the pneumatic hand. Pick a command to see the motion it predicts.

The forward pass, click by click
The forward pass of the Jacobian field: an image is turned into a Jacobian field, which maps a robot command to the 3D motion of points on the robot
  1. From a single image, a neural network reconstructs a 3D Jacobian field.
  2. Take a few points on the robot's surface.
  3. Now take a robot command.
  4. The Jacobian field maps the command to the 3D motion of those points.
The forward pass of the Jacobian field. It plays on its own; click it, or use the arrow keys, to step through by hand.
Move one commandclick a slider
Input image
One camera image of the pneumatic hand
Input command
Predicted 3D motion
The Jacobian field. Each color marks the part of the hand that one command moves.
From one image of the pneumatic hand, the Jacobian field predicts how each command moves the hand in 3D.

Training from random commands

Training needs no kinematic model and no demonstrations. We send random commands and film the robot with a few cameras. The model only watches how the robot looks and how it moves. After training, one camera is enough.

The training objective uses only those recordings. From a single image, the network reconstructs a Jacobian field and a radiance field. Given the command, the Jacobian field gives a 3D motion field. We volume-render both fields into a camera, and supervise the renders with the images and with the optical flow between two consecutive frames.

The training objective, click by click
The training objective: random commands, multi-view video and optical flow supervise a Jacobian field and a radiance field through volume rendering
  1. Start from a sequence of random control inputs.
  2. Take two consecutive time steps.
  3. At both, several cameras see the robot.
  4. Take each camera in turn.
  5. For each camera, we compute the optical flow between its two images.
  6. Now take a single image.
  7. From it, a network reconstructs a Jacobian field and a radiance field.
  8. Given the command, the Jacobian field gives a 3D motion field.
  9. We volume-render both fields into a camera.
  10. And we supervise the renders with the RGB images and the optical flow.
Training with random commands and multi-view video. Rendered color and motion are supervised by the images and by optical flow.

Why render at all? If we give the model depth as input and drop the rendering, its flow error is 7.5 times higher, and its Jacobians blend several actuators together. Our explanation is that without rendering into the other cameras, it gets no multi-view supervision.

And why predict a Jacobian, rather than the motion directly? With the same data and backbone, a model that predicts motion directly still fits the training data but falls apart on new commands. In simulation, its error on unseen commands is 26 to 57 times higher. On the real Allegro hand, its predicted motion smears across the whole hand.

A model that predicts motion directly, compared with the Jacobian field, on held-out samples of the real Allegro hand.

Different bodies, the same visual interface

We tested the same recipe on very different bodies: the 3D-printed pneumatic hand, the 16-DoF Allegro hand, a soft, compliant wrist made of handed shearing auxetics (HSA), and a low-cost 3D-printed arm. Each one learns its own Jacobian field from its own experience. For every body, the model recovers the 3D shape and the kinematic structure from vision alone.

Pick a body
Input image
Pneumatic hand, input image
Predicted color
Predicted depth
Predicted Jacobian field
Input image
Predicted color
Predicted depth
Predicted Jacobian field
Input image
HSA platform, input image
Predicted color
Predicted depth
Predicted Jacobian field
Input image
Poppy arm, input image
Predicted color
Predicted depth
Predicted Jacobian field
Each body learns its own model from its own experience. In the Jacobian field, each color is one command channel, and shows which parts of the body that command moves. HSA: handed shearing auxetics, a soft, compliant wrist.

Control

With a Jacobian field, control becomes an optimization. We specify the motion we want, in 2D or 3D, and solve for the command whose predicted motion matches it. Gradient-based optimization finds that command in about half a second. The second figure below shows the optimization running on the pneumatic hand.

From desired motion to a command
A desired motion becomes a control objective: gradient-based optimization finds the command, which is then executed on the robot
  1. Start from the Jacobian field of the current image.
  2. Specify the motion we want, in 2D or 3D. Gradient-based optimization finds the command whose predicted motion matches it, in about half a second.
  3. Executing that command on the robot produces the motion.
A desired motion becomes a control objective.
Desired motion on the pneumatic hand
Desired motion
Optimization
The command found by the optimization
Output command

For longer motions, we close the loop. At every step, the robot looks at the current image, solves for a small command toward the next waypoint, executes it, and looks again.

Closed-loop tracking
Closed-loop trajectory tracking: the arm draws a square in four segments, and each segment ends within 8 mm of its target
  1. The low-cost arm draws a square in four segments. White is the reference trajectory, color is the completed motion.
  2. In each segment, the distance to the target drops to a few millimeters.
Closed-loop tracking of a square, from Fig. 4e of the Nature paper.

The same recipe works on very different bodies. The pneumatic hand closes its fingers while part of it is hidden behind objects. The low-cost arm traces the letter M in the air. The soft wrist rotates while carrying a 350 g load: the load tilts it at rest, and the visual feedback still drives it to its goal, about 7 mm from the target.

Pick a body
Closing the fingers under partial occlusion
Desired motion
Execution
Counter-clockwise rotation while carrying a 350 g load
Desired motion
Goal configuration
The HSA platform at its goal configuration, carrying the load
Tracing the letter M in the air
Desired motion
Achieved trajectory
The target is only a motion in the world. Each body's Jacobian field turns it into that body's commands.

A Jacobian field is a body program

Since each body learns its own model from its own experience, the hardware no longer has to be designed around our ability to model it by hand. Stepping back, the Jacobian field turns a motion plan into one body’s commands. I call this a body program. So far the motion plan came from us, because there was no planner. The second half of the talk was about learning that level too.

Two levels, click by click
A two-level picture: planning programs at the high level, shared across bodies, and one learned body program per body at the low level
  1. For each body, a Jacobian field turns a motion plan into that body's commands.
  2. I call that a body program. It takes what the camera sees and a motion plan, and returns actuator commands. Each body learns its own.
  3. But where does the motion plan come from? So far, from us: there was no planner.
  4. Next, we learn that level too. A planning program takes an instruction and proposes the motion.
Body programs at the low level, one per body, below a shared planner.

Planning in video

VERA is a visual planning program. It plans in video, acts through a body program, and replans from what the cameras see.

VERA, click by click
VERA: a video world model generates visual futures from a text prompt and observations, a Jacobian inverse dynamics model turns them into actions, and execution feeds back new observations
  1. Given a language instruction and the current observations, a video model generates a video plan.
  2. A body program, our Jacobian inverse dynamics model (J-IDM), turns each pair of planned frames into an action.
  3. The robot executes the first few actions, observes what happened, and plans again.
VERA plans in video, acts through a body program, and closes the loop.

The planner is Wan, a pretrained 14-billion-parameter video diffusion transformer. We did not change its architecture. Camera views are tiled into one wide frame, and we post-train the model on robot videos from DROID, only to predict future frames. There is no action head. Actions come from the body program.

The instruction decides which plan the model generates. Pick one below: the observed frames stay the same, and only the words change.

Pick an instruction
Observed frames
External 1External 2Wrist
Three camera views of the scene, tiled into one wide frame
Future frames start as noise
“Approach the cup”
Pretrained
Wan
video diffusion transformer
14B parameters
No architecture change.
No action head.
Generated video plan
The gripper moves to the cup.
All three plans start from the same camera images. Only the instruction changes. Frames outlined in red are the observed ones.

The body program starts from VGGT, a pretrained multi-view geometry transformer. We remove its geometry heads and add a Jacobian head that predicts, for every pixel, how each command channel moves that pixel. We train it like NJF, from known commands and observed optical flow, but in 2D pixels and without rendering. At test time we invert it: the planned pixel motion goes in, and a command comes out.

Command channel
Cameras
Three camera views of a real Panda arm
Pretrained
VGGT
multi-view geometry transformer
NewJacobian head
depth · point map
camera · tracks
VGGT's own heads, removed
Per-pixel Jacobianone 2 × K matrix per pixel
Predicted Jacobian for five command channels in three camera views
Each column is one command channel, each row one camera. Brighter pixels move more when that command changes; arrows show the direction.
The body program for a real Panda arm. Hover or click a column to look at one command channel. The Jacobian panel is from the VERA paper (Fig. 3).

Why start from a pretrained video model? Internet video already teaches a lot about motion and interaction. When we trained the planner from scratch on the same robot videos, its held-out prediction error was 3 times higher (0.40 vs. 0.13). One way to picture this, borrowing from Neural Thickets, is below. The picture is an analogy, not something we measured.

An analogy
A schematic of weight space: regions of robot behaviors sit around the pretrained video model, and post-training takes a short step toward them
  1. Internet video already teaches a lot about motion and interaction.
  2. Many robot behaviors sit close to the pretrained weights.
  3. Post-training on robot video is a short step toward them.
Robot behaviors near the pretrained weights. An analogy, after Neural Thickets.

What does post-training keep? In our tests, the planner kept working with new camera viewpoints, new object arrangements, camera-angle changes, and lighting changes. These are examples, not a controlled comparison. My intuition is that we adapt behavior on top of a broad visual foundation, instead of learning each situation from scratch.

Does the Jacobian structure still matter once there is a planner? We compared three heads with the same backbone, model size, and data. The Jacobian head stays accurate as we add degrees of freedom, and it needs less data. In closed loop in simulation, it succeeds 70% of the time on the 16-DoF hand and 94% of the time on the 7-DoF arm, where a UniPi-style direct inverse dynamics model succeeds 0% of the time on both.

Action reconstruction error as the degrees of freedom (left) and the number of training samples (right) grow. Lower is better.

On real robots

A few runs from our evaluations. These are examples, not controlled comparisons. First, place the yellow cube on top of the lego block. On the left is the generated plan, on the right what the real robot did.

Generated plan (left) and real execution (right).

The same planner also drives a 16-DoF hand. In simulation, only the prompt changes. Here the body program outputs 16 joint commands.

Prompt
Closed-loop rollouts in simulation, each seen from three cameras. The full prompts end with “around the vertical axis”.

And on the real hand, with the clockwise prompt. The left two panels are the plan from the video model, and the right two show the real hand executing it.

Generated plan (left two) and real execution (right two), from two cameras.

Push the blue button. At the start, only one of the external cameras can see it, and the orange button is a distractor. The wrist camera first looks at the orange button, then finds the blue one and presses it.

Wrist camera, external camera 1, external camera 2.

Finally, we moved the external camera between the two settings below, and asked the robot to place the yellow cube on the white plate. Halfway through, someone moves the plate. In this run, the robot follows the plate and still places the cube on it.

External camera 1, external camera 2, wrist camera.

Where should a shared model meet the body?

Video models are becoming robot policies in industry too, for example Rhoda AI’s Direct Video-Action Model and NVIDIA’s GR00T 2 world action model. So the question is less whether to use a video model, and more where the shared model should meet the body.

The usual way to share one model across robots is a generative action model with one large action placeholder. RDT-1B defines a 128-dimensional action vector whose slots have fixed physical meanings: arm joints, grippers, end-effector pose, and the base. Each robot fills in the slots that match its actions, and the rest are padded. Our Allegro hand has 16 finger joints and our pneumatic hand has 15 pressure channels, and neither has a slot. π0 instead zero-pads every robot’s actions to the size of the largest robot, so the same output means different things on different robots. Either way, the shared model has to output each body’s own coordinates.

We do the opposite. The shared program never outputs an action. It outputs effects in the world, a video plan, and each body gets its own body program, learned from its own experience. Try each of our four bodies both ways.

Pick a body
Standard approachgenerative action modeling
High level
Generative action model
shared across robots
Action placeholderone vector, every robot
arm joints
gripper
end-effector
base
reserved
Low level
Fit every robot into slots
slots defined ahead of time
Articulated arm6 motor commands
This talkembodied programming
High level
Visual planning program
shared across bodies
Effects in the worldgenerated video plan
“Place the yellow cube on top of the lego block”→
Low level
Body programs
constructed per body
body program →
Embodiment generalization need not be a distinct learning problem.
Pick a body. On the left it has to fit a placeholder that was defined ahead of time (a 24-slot schematic, not RDT-1B's exact 128-slot layout). On the right the same video plan goes through that body's own body program.

This is the argument of the talk. Embodiment generalization need not be a distinct learning problem. Much of the problem comes from describing behavior in the coordinates of one particular body. If we describe behavior by its effects in the world, shared programs can act through different bodies, and each body can learn its own mapping from experience.

Looking forward

Everything above takes an instruction and moves a body. So the whole system can be one tool. A multimodal agent, such as Claude, could call act("put a plate at each seat") the way it calls any other tool, and break a long task into such calls. Programs have a recursive structure: a whole system today becomes one call inside a larger program.

A program that calls this talkclick a line, or
High level
Multimodal agent
e.g., Claude
# goal: set the table for two
tool call
Low level
Toolkits
= this talk is one instantiation
act(instruction)
Visual planning program
Effects in the world
Body programs
Programs have a recursive structure: a whole system becomes one call in a larger program.
A sketch of where this could go, not something we have built. Each act(...) call runs everything in this talk: a video plan, then a body program.

We have not built this part yet. NVIDIA already describes an agent that controls GR00T 2 as a tool through an API, so I expect this to move quickly. This is where I think embodied programming is going in 2027.

Acknowledgments

Thank you to the Carney Institute for hosting me, and to everyone who came. Neural Jacobian Fields is joint work with Annan Zhang, Boyuan Chen, Hanna Matusik, Chao Liu, Daniela Rus, and Vincent Sitzmann. VERA is joint work with Evan Kim, Xingjian Bai, Tong Zhao, Tao Pang, Max Simchowitz, and Vincent Sitzmann. I am advised by Vincent Sitzmann and Josh Tenenbaum.

Neural Jacobian Fields: Paper · Project page · Code

VERA: Paper · Project page · Code

Citation

Please cite this post as:

Sizhe Lester Li. "Embodied Programming Through Vision." November 2026. https://sizhe-li.github.io/blog/2026/embodied-programming-through-vision/

or, in BibTeX:

@misc{li2026embodiedprogramming,
  title        = {Embodied Programming Through Vision},
  author       = {Li, Sizhe Lester},
  year         = {2026},
  month        = nov,
  howpublished = {\url{https://sizhe-li.github.io/blog/2026/embodied-programming-through-vision/}},
  note         = {Blog post}
}