Fixing weak language grounding in a robot policy’s weights

Same goal, different words

Robot policies are sensitive to how an instruction is worded [1, 2]. In an earlier study we gave three world-action models the same placement goals in different words [3]. When the instruction described the goal from the reference object’s side instead of the moved object’s side, stable placements fell by 12.2 points across 864 rollouts.

That study measured the gap. This post asks whether we can close it inside the policy.

Our running example is π0.5, an open vision-language-action policy (VLA) [4, 5], on a task from RoboLab, a simulated benchmark [6]. The instruction is “Put the cube then the banana in the bowl.” The robot picks up the cube and lets go of it on the way to the bowl. It completes the task about one time in five.

To find out why, we recorded 21 moments where the robot let go of the cube. We replayed each moment and changed only the instruction.

Renaming the cube changed nothing. Putting “in the bowl” right after “the cube” removed almost every drop. So the policy was attaching “in the bowl” to the banana, and while it held the cube it acted as if it held the wrong object.

The policy has the skill. Given “Put the cube in the bowl then put the banana in the bowl”, the same weights completed 22 of 32 runs. Given the original wording they completed 3.

We call this weak language grounding [7, 8]. The policy can do the task, but it grounds the instruction to the wrong object or relation [9].

A clear wording works. Can we make the original wording work too, without changing a word?

Edit the weights, keep the words

We cannot always reword an instruction. Instructions come from people and from other software.

Each of these policies has a component that grounds the instruction. In a VLA it is the VLM backbone, the vision-language model that takes in the camera images and the instruction [10, 11]. In a world-action model it is the text pathway, the layers that carry the instruction tokens [12]. A second component generates the actions. It is an action expert [5] or an action head [13].

The clear wording already produces the right activations in the component that grounds the instruction. So we copy them. We record moments where the robot fails, and we replay each one with both wordings. Then, for each layer, we solve for the weight change that makes the original wording produce the clear wording’s activations.

This is context distillation [14, 15] in closed form. Each layer takes one least-squares solve, the same update used to edit facts in language models [16, 17]. We run no training and collect no new data. The robot still receives the original instruction.

We work by weight-space programming [18]. An LLM editor, here Claude Opus, reads and writes the weights with tools instead of an optimizer. Before any robot run, the editor checks each edit offline. It replays the recorded moments through the edited weights and looks at the actions the policy plans. Three simpler edits failed this offline check before the one above passed.

How we score an edit

A run is one attempt at a task. Every run of a task begins from the same scene and has its own random seed. We build and choose an edit on practice runs. We score it on held-out runs, whose seeds played no part in building it.

The simulator is noisy. When we ran one policy twice on the same seeds, about 30% of the runs ended differently on one task and almost 60% on another. So we compare the edited and the unedited policy on the same seeds, and we trust a difference only when it is larger than this noise.

We also compare every edit with a random edit of the same size. It changes the same layers with the same rank and singular values, in random directions. Concurrent work uses the same control [19].

It works on a VLA with an action expert

π0.5 pairs a PaliGemma VLM backbone with a separate action expert. We wrote the edit into the VLM backbone and scored it on 128 held-out runs. The unedited policy completed 27 of them, and the edited policy completed 106. The edit fixed 81 runs and broke 2.

A random edit of the same size did nothing. The edit also recovers most of what the clear wording gives. On the same 64 runs, the edited policy completed 50 with the original wording, and the unedited policy completed 57 with the clear wording.

A compact version keeps most of the effect. It changes only what the VLM backbone computes at the image tokens in layers 8 to 16, and it can be stored in 0.16% of the parameter count.

The same edit helped two other tasks

We ran the edit unchanged on 15 other tasks. Two improved: “Pick up the eyeglasses” and “Put the cube to the left of the bowl”. On the other 13 we saw no clear change. Those tasks had 16 to 32 runs each, so we could have missed a small change.

The edit does not make the policy better at spatial language in general. “Put the cube to the right of the bowl” still never works. Our guess is that the edit strengthens how the policy ties a destination, such as “in the bowl” or “to the left of”, to the object in the gripper.

All of this is one policy. Does the same edit work on policies that are built differently?

Does it carry over to other architectures?

We tested two more architecture types, with one released model of each.

A world-action model [20] predicts the video of what will happen and the robot’s actions together, in one network [21]. Our instance is Cosmos3 Edge [12]. Its instruction tokens run through their own text pathway.

A VLA on a reasoning backbone grounds the camera images and the instruction in a reasoning VLM, and it generates action chunks with an action head [22]. Our instance is GR00T N1.7 [13], with a Cosmos-Reason2 backbone and a flow-matching [23] diffusion-transformer head [24].

We ran all three on the same RoboLab tasks in Isaac Sim [25], and we paired each edit with a random edit of the same size. We start with the world-action model.

In a world-action model, the edit works as well as the clear wording

The world-action model has the same kind of failure on another task. Here are two wordings of one goal:

Both wordings state one spatial relation, each from a different object’s side. With the second wording the model completed 31 of 32 practice runs. With the first it completed none.

We built the edit from this pair and wrote it into the text pathway. On 64 held-out runs the edited model completed 56, with the original wording. The unedited model completed the same number when we gave it the clear wording. The unedited model and the random edit completed none.

So the edit carries over to this world-action model. Three things decide whether it works: where the edit goes, how the two wordings line up, and whether the policy has the skill at all.

The edit works best where the instruction is grounded

We wrote the same edit into each component of each architecture.

On π0.5 we also tried three variants. One restricted the VLM edit to the image tokens. One edited the VLM backbone and the action expert together. One was a gated edit of the action output projection, written by an LLM editor that never saw our edit. Each of them fixed part of the failure.

In both models where the edit works, it helped most in the VLM backbone or text pathway. In π0.5’s action expert it helped less. In the world-action model’s video/action pathway it did nothing.

We think the reason is cause and symptom. The clear wording changes the VLM backbone or text pathway first, and the components downstream follow. An edit there copies the cause. An edit in the action expert has to copy activations that also depend on where the robot is, so it copies a symptom. On π0.5’s action output projection, the edit fixed nothing when it acted at every step. It helped only behind a gate that switches it on at the right moments, as in activation steering [26].

Interpretability work points the same way. In π0.5, information about the instruction reaches the action expert through a narrow band of layers [27]. Our offline replays agree. In the world-action model, the text-pathway edit reproduces almost all of the clear wording’s effect on the planned actions. The video/action-pathway edit reproduces less than a quarter.

The two wordings have to line up

The edit copies the clear wording’s effect onto the original wording token by token. Each token’s input is a key, as in model editing [17]. On the mustard task the two wordings have the same length and swap words in place. Every token has a counterpart, and the edit works.

When the clear wording adds words, the added tokens have no counterpart. The edit failed this way on the world-action model’s Cube left of bowl and on the reasoning-backbone VLA’s Cube then banana.

We wrote this rule down before we tested two new pairs of wordings. The first pair has equal length: Cube left of bowl, worded bowl-first and cube-first. The edit lifted the world-action model from 1 to 12 of 32 held-out runs, far above a random edit. The second pair has unequal length, and its edit failed the offline check, as the rule predicts.

π0.5 is the exception. Its clear wording adds words, and the edit still works. π0.5’s VLM backbone attends over the image tokens and the instruction together, so the clear wording’s effect also reaches the image tokens. Both wordings share those tokens, and the edit copies the effect there. π0.5 is known to hold spatial information at its image-token positions [27].

The policy must already have the skill

The edit copies a behaviour that the policy already shows under some wording. It did not help when no wording produced the behaviour.

We tried three other π0.5 failures, with Claude Opus and GPT-6 Astra as the LLM editors. In one the robot grabs a lookalike block. In another the box is too wide for the robot’s usual grasp. In the third a mustard bottle slips out of the gripper. The editors wrote dozens of edits, and none helped. The policy can tell the lookalike blocks apart, but no input we tried made it act on that. The box needs a grasp the policy never learned, and the bottle slips.

We also changed one bias value so that the gripper closes more readily. Across seven tasks the policy completed exactly as many runs with the change as without it.

The reasoning-backbone VLA fails for the same reason. Without any edit, GR00T N1.7 completes only 3 of the 9 tasks at all. A clear wording helped on one task, Cube then banana, and the gain was within noise. We copied that wording into the VLM backbone and into the action head. With either edit the robot completed none of 32 practice runs, while random edits of the same size left it about where it started.

The cost: the edit overgeneralises the relation

We ran the world-action model’s mustard edit unchanged on every other task. Two tasks broke. The edit was built from a sentence pair that turns “left of” into “right of”, and both broken tasks have instructions that say “left”. On one of them the robot now carries the mustard to the wrong bin.

The edit changed how the text pathway grounds a spatial relation, and the change reached other instructions that use the relation. Model-editing work calls this a loss of specificity or locality [28], or a ripple effect [29]. Directional words are also where π0.5 and GR00T N1.7 are most sensitive [30].

π0.5’s edit behaved differently. Run the same way, it helped two tasks and broke none that we could detect.

Protecting other instructions did not help in our tests

We rebuilt the mustard edit twice with constraints that tell the solve to leave other instructions unchanged [17]. The first rebuild protected 65 unrelated instructions. The second protected 24 instructions from other tasks that use the same spatial relations. Both rebuilds kept the fix, and both still broke a task that shares the relation.

A third rebuild copied only the words that both wordings share. In replays it barely touched the other tasks, and it also lost the fix.

These tests were small. Each used 16 practice runs in one setting, with no control. Methods that enforce preservation exactly, by projecting the update away from the protected keys [31], might do better. We did not test them.

A second edit did not undo it

The world-action model also fails Cube left of bowl because of weak grounding. A more specific wording about doubles its success on practice runs. But that wording adds words, and its edit failed the offline check. Stacked on the mustard edit it failed again, so neither edit ran on the robot.

Can an LLM find these edits on its own?

We gave seven fresh Claude Opus editors the same tools and the same small budget. Each got one π0.5 task with a different kind of failure. No editor saw our edit or another editor’s work. We scored each edit they submitted on 64 held-out runs.

The editor on the cube task found the grounding failure by itself. It replayed the drops with a clear wording, as we had. Then it wrote a different and much sparser edit that changes 1,024 weights. That edit raised success from 11 to 27 of 64, which is less than our edit and more than noise.

The other four editors submitted nothing, which the protocol asks for when no edit holds up. On Cube left of bowl the fix broke grasping. On the three-cube stack the edit failed its offline check. On the lookalike block no wording helped. On the flat box the grasp held and the box still slipped. Every editor’s notes and code are in the repository.

Putting it together

What this means

Some robot failures come from weak language grounding. The policy has the skill, and the instruction fails to trigger it. A policy can even encode the instructed target without that target controlling the action [32].

When a clear wording works, a closed-form edit can copy its effect into the weights. In our tests the edit worked when three things held. The policy had the skill. The edit went into the component that grounds the instruction. And the two wordings lined up token by token, or they shared image tokens that carried the effect.

The edit can also change other instructions that use the same relation. So we check every edit on tasks it was not built for.

Our rule of thumb is to edit when some wording already makes the robot do the right thing, and to train when none does. Two outside reports of large gains from weight edits fit this rule. One edited checkpoints that were already fine-tuned on each task [33]. The other built its updates from differences between trained models [34]. We did not reproduce either result.

Limits

Simulation only. Every number comes from RoboLab tasks in Isaac Sim. We ran nothing on a physical robot.

One scene per task. Held-out runs change the random seed and keep the scene, so they do not test new object positions.

Small samples. With 64 runs per comparison we can detect a change of about 19 percentage points. With 16 to 32 runs we can detect only larger ones.

One released model per architecture type. The results describe π0.5, Cosmos3 Edge and GR00T N1.7. Each model has its own cameras and action-chunk length, so compare what an edit fixes or breaks within a model, and do not compare success rates across models.

The edit needs a wording that works. We found each clear wording by trying rewordings.

The edit is low-rank and not sparse. It changes many weights by a small amount.

Cosmos3 Edge’s wiring. We read it from the code that serves a sibling model, Cosmos3 Nano [12]. In that code the video/action pathway reads the instruction only through the text tokens’ keys and values.

Try it

weightsmith is a harness for weight-space programming on robot policies. An LLM editor, or you, reads, probes and writes π0.5’s weights through one tool. The harness scores every edit against the unedited policy on held-out runs.

git clone https://github.com/adeeb10abbas/weightsmith && cd weightsmith
pip install -e '.[mcp]'
weightsmith --help

Start with the README. The benchmark has the rules for submitting an edit, and the leaderboard has every result so far. The explorer shows every π0.5 comparison run by run. All numbers and records has every table, the protocol and more clips for the three architectures.

References

1. Fei, Wang, Shi, Dai, et al. LIBERO-Plus: A Progressive Robustness Benchmark for Visual-Language-Action Models. CVPR 2026. arXiv:2510.13626
2. Kim, Kim, Kang, Kim, Jung. LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models. EMNLP 2026. arXiv:2603.28301
3. Abbas, Seraj, Toghi, Padır. Same Goal, Different Words: Spatial Instruction Sensitivity in World-Action Models. arXiv 2026. arXiv:2610.16242
4. Physical Intelligence, Black, Brown, et al. π0.5: a Vision-Language-Action Model with Open-World Generalization. arXiv 2025. arXiv:2504.16054
5. Black, Brown, Driess, Esmail, et al. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv 2024. arXiv:2410.24164
6. Yang, Dagli, Zook, Hadfield, et al. RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies. RSS 2026. arXiv:2604.09860 · doi:10.15607/RSS.2026.XXII.096
7. Shridhar, Manuelli, Fox. CLIPort: What and Where Pathways for Robotic Manipulation. CoRL 2021. arXiv:2109.12098
8. Ahn, Brohan, Brown, Chebotar, et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. CoRL 2022. arXiv:2204.01691
9. Fang, Feng, Jing, Liu, et al. When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs. CoRL 2026. arXiv:2602.17659
10. Brohan, Brown, Carbajal, Chebotar, et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. CoRL 2023. arXiv:2307.15818
11. Kim, Pertsch, Karamcheti, et al. OpenVLA: An Open-Source Vision-Language-Action Model. CoRL 2024. arXiv:2406.09246
12. NVIDIA (Aditi, Agarwal, et al.). Cosmos 3: Omnimodal World Models for Physical AI. arXiv 2026 (technical report). arXiv:2606.02800
13. NVIDIA: Bjorck, Castañeda, et al. GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. arXiv 2025. arXiv:2503.14734
14. Askell, Bai, Chen, Drain, et al. A General Language Assistant as a Laboratory for Alignment. arXiv 2021. arXiv:2112.00861
15. Snell, Klein, Zhong. Learning by Distilling Context. arXiv 2022. arXiv:2209.15189
16. Meng, Bau, Andonian, Belinkov. Locating and Editing Factual Associations in GPT. NeurIPS 2022. arXiv:2202.05262
17. Meng, Sen Sharma, Andonian, Belinkov, Bau. Mass-Editing Memory in a Transformer. ICLR 2023. arXiv:2210.07229
18. Takehi, Teoh, Zhang, Khattab. Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. Technical blog, 2 Nov 2026. rikiyat.github.io/blog/weight-space-programming
19. Zhang, Wang, Lin. Can LLMs Debug Robots by Editing Their Weights? arXiv 2026. arXiv:2611.09376
20. Shen, Zhang, Liao, Li, et al. World Action Models: A Survey. arXiv 2026. arXiv:2606.20781
21. Kim, Gao, Lin, Lin, et al. Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning. arXiv 2026. arXiv:2601.16163
22. Chi, Xu, Feng, Cousineau, Du, Burchfiel, Tedrake, Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS 2023 (IJRR 2024 extended). arXiv:2303.04137
23. Lipman, Chen, Ben-Hamu, Nickel, Le. Flow Matching for Generative Modeling. ICLR 2023. arXiv:2210.02747
24. Peebles, Xie. Scalable Diffusion Models with Transformers. ICCV 2023. arXiv:2212.09748
25. NVIDIA (Mittal, Roth, et al.). Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning. arXiv 2025. arXiv:2511.04831
26. Turner, Thiergart, Leech, Udell, Vazquez, Mini, MacDiarmid. Steering Language Models With Activation Engineering. arXiv 2023. arXiv:2308.10248
27. Swann, Papillon, Jacobs, Boppana, et al. Causal Analysis of Robot Foundation Models Enables Efficiency Gains. arXiv 2026. arXiv:2610.20750
28. Wang, Zhu, Liu, Zheng, et al. Knowledge Editing for Large Language Models: A Survey. ACM Computing Surveys 2024 (arXiv 2023). arXiv:2310.16218 · doi:10.1145/3698590
29. Cohen, Biran, Yoran, Globerson, Geva. Evaluating the Ripple Effects of Knowledge Editing in Language Models. TACL 2024. arXiv:2307.12976 · doi:10.1162/tacl_a_00644
30. Wulff, Cangelosi. Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding. arXiv 2026. arXiv:2610.10178
31. Fang, Jiang, Wang, Ma, et al. AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models. ICLR 2025. arXiv:2410.02355
32. Jiang, Cao, He, Deng, et al. Encoded but Not in Control: Revealing the Grounding Gap in Vision-Language Robot Policies. arXiv 2026. arXiv:2610.06235
33. Guo, Xie. Weight Editing across Robot Policies: VLA, WAM, and Humanoid Control. Experiment thread on X, November 2026, author-reported. x.com/Lingxiao234
34. Gupta, Rohilla, Prashant, Zhu. Can Frontier LLMs Fix Robots by Editing Their Weights? Technical blog, November 2026. codnay.github.io

Citation

Ali-Adeeb Abbas and Esmaeil Seraj. “Distilling Instructions into Weights: Closed-Form Edits Fix a Robot Policy’s Misreading.” 2026. arXiv preprint, forthcoming.

@misc{abbas2026distilling,
  title  = {{Distilling Instructions into Weights: Closed-Form Edits Fix a Robot Policy's Misreading}},
  author = {Abbas, Ali-Adeeb and Seraj, Esmaeil},
  year   = {2026},
  url    = {https://ali-adeeb.com/weightsmith/},
  note   = {arXiv preprint, forthcoming}
}

We measured every result here ourselves on paired simulator runs, except where we say that a result is a guess or comes from another group. The exact numbers and statistical tests are in the paper and the result files. Each π0.5 video shows one run where only the edited robot succeeds, at twice real speed. This work builds on weight-space programming (Takehi et al.) and uses Physical Intelligence’s open π0.5 model and the RoboLab tasks in Isaac Sim.