LLM AGENTS HIT FULL PATCH COVERAGE IN 30% OF PRS—YET REGRESS IN MULTI-TURN EDITS
LLM-assisted PR augmentation can reliably raise patch-level assurance: a study of ChaCo achieves full patch coverage on 30% of PRs at ~$0.11 each with high revi...
LLM-assisted PR augmentation can reliably raise patch-level assurance: a study of ChaCo achieves full patch coverage on 30% of PRs at ~$0.11 each with high reviewer acceptance and real bug finds coverage gains 1. But multi-turn agent editing remains brittle—Mr Dre shows deep research agents regress on ~27% of previously correct content and citation quality even while applying >90% of requested edits revision regressions 2. Benchmarks in embedded development highlight that tight tool feedback loops matter: EmbedAgent finds low baseline pass rates that improve with RAG and compiler feedback domain benchmark 3, while AgenticPruner (using Claude 3.5 Sonnet) and Intervention Training demonstrate structured feedback can improve deployment efficiency and reasoning accuracy (agentic pruning4, reasoning self-intervention5).
-
Adds: ChaCo method, results across SciPy/Qiskit/Pandas, cost, human acceptance, and bug discoveries. ↩
-
Adds: Mr Dre evaluation, 27% regression metric, and limits of prompt/sub-agent fixes in multi-turn revision. ↩
-
Adds: first embedded-dev benchmark (EmbedBench), model pass@1 gaps (e.g., ESP-IDF vs. MicroPython), and gains from RAG + compiler feedback. ↩
-
Adds: agentic multi-agent pruning guided by Claude 3.5 Sonnet, MAC-targeted compression with accuracy/speedup results. ↩
-
Adds: Intervention Training approach for step-level credit assignment, ~14% reasoning accuracy gain on IMO-AnswerBench. ↩
Patch-focused test generation can cut regressions cheaply, but unsupervised multi-turn agents risk degrading correct work.
Tool-augmented loops (RAG, compiler/test feedback, analysis agents) make LLM workflows more predictable and production-ready.
-
terminal
Pilot a CI bot that proposes tests only for changed lines, auto-runs them, and opens a test-only PR for human review.
-
terminal
Gate multi-turn agent edits with diff-aware guards (locked sections, citation checks) and rollbacks when quality drops.
Legacy codebase integration strategies...
- 01.
Add a context-aware test-gen job on PR diffs and track merge rate, flake rate, and escaped bug count.
- 02.
For niche stacks, wire RAG over internal docs and feed compiler/linter output back into prompts to stabilize codegen.
Fresh architecture paradigms...
- 01.
Design tests with reusable fixtures and data generators to maximize LLM test synthesis quality and maintainability.
- 02.
Prefer stacks with richer tool feedback and better LLM affordance (e.g., MicroPython over ESP-IDF) if automation is a priority.