DEEP_DIVE_STORY.LNK

New research isolates which “oracle” hints actually move SWE-bench agent scores, explaining why headline results often don’t match real coding impact.

Microsoft and Georgia Tech introduced Oracle-SWE, a method that extracts idealized signals—like failing test reproduction, regression tests, edit locations, execution context, and API usage—from SWE benchmarks to quantify each signal’s contribution to agent success. It evaluates how much a base agent improves when fed these signals, separating model ability from scaffolding.

That framing helps decode the hype cycle. Claims that GLM-5.1 “topped SWE-Bench Pro” have circulated, alongside reports that it felt weak for day-to-day coding; see the contrasting read in this analysis. A broader critique argues benchmarks are being Goodharted into marketing targets rather than reliable measures of value essay.

Net: treat public scores as directional. Run ablations to see which signals your stack can realistically provide and whether those gains transfer to your repos.