Every comparison

Each row is one closed-loop comparison from this work or the companion Astra study: success with an intervention minus success without it, on the same simulator seeds.

Back to the article·Data (JSON)

Failure type Editor Seeds

Hover, tap or tab to a row for its counts. Intervals are 95% paired bootstrap intervals computed in the browser from each row’s seeds and fixed and broken counts (2,000 resamples, fixed seed); the paper’s intervals are authoritative. GPT-6 Astra rows come from that study’s results and ran on a separate runtime, so they are not pooled with ours.

The same comparisons as a table.
Failure type	Task	Intervention	Editor	Seeds	Without	With	Fixed / broke	Change, points

Use it as a benchmark

The benchmark gives an editor one tool, owsp, to read weights, capture activations on recorded inputs and run paired practice rollouts, with drivers for Claude Code and the Codex CLI. An operator then scores the editor’s one frozen edit on held-out seeds with an exact McNemar test. Entries are on the leaderboard; setup is in docs/SETUP.md.