Same goal, different words
6 November 2026 · A note on our CoRL 2026 workshop paper
If a robot understands a goal, the wording should not matter. "Put the cube to the left of the bowl" and "the bowl should have the cube on its left" ask for the same world. We wanted to know whether current world-action models treat them that way.
What we did
We took three world-action models (Cosmos3 Nano, Cosmos3 Edge, and FLUX 3 Action), twelve spatial goals across four simulated tabletop scenes, and eight starting states for each. Every goal was written three ways: a direct instruction, a target-first phrasing that names the object to move before the reference, and a reference-first phrasing that names the reference object before the object to move. Everything else was held fixed. That gives 864 rollouts, each one paired with the same rollout under the other two wordings.
We scored each rollout on whether the requested arrangement was ever reached, whether it was still true at the end of the episode, and which object actually moved.
What we found
Reference-first wording was placed less often than target-first wording in every model, by about twelve points on average.
| Model | Target-first | Reference-first | Gap |
|---|---|---|---|
| Cosmos3 Nano | 50.8% | 36.2% | −14.6 |
| Cosmos3 Edge | 31.0% | 22.9% | −8.1 |
| FLUX 3 Action | 56.0% | 42.2% | −13.8 |
The pooled gap is 12.2 points, with a 95% confidence interval of 8.6 to 15.4. That is not a property of one checkpoint. It shows up in all three.
The failures are also informative. For the goal "cube behind the bowl", every reference-first rollout that satisfied the relation did so by moving the bowl rather than the cube. The arrangement was right and the behavior was wrong. A success metric that only checks the final relation would have counted those as wins.
Placement is also fragile in time. Of 864 rollouts, 383 reached the requested arrangement at some point, and only 191 still had it at the end of the episode. Models knock things over, keep moving after the goal is met, or put the object down and pick it up again.
To see where the sensitivity lives, we probed the language backbones on text alone. On reference-first phrasings they sometimes reversed the relation outright, answering "left of" for an instruction whose correct reading is "right of". Adding the image did not reliably help, and in some cases it broke a reading the text encoder had gotten right.
What we take from it
These models are not indifferent to wording, and the way they fail is not random. The results argue for three small habits in evaluation: test consistency across equivalent instructions rather than one phrasing, verify that the intended object moved rather than only the final relation, and check whether the arrangement persists to the end of the episode rather than whether it was ever true.
The study is limited to simulation, three checkpoints, and tabletop placement. We do not claim a ranking of models, and the exact numbers will move with new checkpoints. The pattern is what we think is worth attention.
The project site has every result from the paper and an explorer for all 864 rollouts. The evaluation code and scene registry are on GitHub. This work was done with Esmaeil Seraj, Behrad Toghi, and Taşkın Padır.