terminal
howtonotcode.com
SWE-Bench Verified logo

SWE-Bench Verified

Term

A framework for evaluating software engineering tools.

article 6 storys calendar_today First seen: 2026-02-03 update Last seen: 2026-03-03 open_in_new Website menu_book Wikipedia

Stories

Showing 1-6 of 6

Coding Benchmarks Shake-up: Qwen 3.5, MiniMax M2.5, and a SWE-bench Reality Check

Open models like Alibaba’s Qwen 3.5 and MiniMax M2.5 post strong coding-agent results, but OpenAI’s audit of SWE-bench Verified shows contamination and flawed tests that can mislead real-world adoption. Alibaba’s Qwen 3.5 family uses a sparse MoE design (397B total/17B active), ships open weights under Apache 2.0, and shows strong instruction following and competitive coding scores in public benchmarks, with setup guidance and comparisons to frontier models detailed in this deep-dive guide [Qwen 3.5: The Complete Guide](https://techie007.substack.com/p/qwen-35-the-complete-guide-benchmarks). MiniMax’s latest model claims state-of-the-art coding and agentic performance, faster task completion, and ultra-low runtime cost (about $1/hour at 100 tok/s), alongside reported scores on coding and browsing evaluations [MiniMax-M2.5 on Hugging Face](https://huggingface.co/unsloth/MiniMax-M2.5). OpenAI, however, reports that many SWE-bench Verified tasks have broken tests and that major models were trained on benchmark solutions, halting its use of the metric and urging caution in interpreting scores [OpenAI Abandons SWE-bench Verified](https://blockchain.news/news/openai-abandons-swe-bench-verified-contamination-flawed-tests). For quick, low-cost trials of multiple “top models,” a short explainer points to an Alibaba Cloud coding plan bundling popular options [This $3 AI Coding Plan Gives You Every Top Model You Need](https://www.youtube.com/watch?v=Qnz7S-5fzWo&pp=ygUXbmV3IEFJIG1vZGVsIGZvciBjb2RpbmfSBwkJrgoBhyohjO8%3D).

calendar_today 2026-03-03
qwen-35 alibaba alibaba-cloud minimax-m25 openai

Agentic coding meets reality: benchmarks expose gaps, runtime tracing narrows them

New evidence shows LLMs still struggle with production-grade observability and cross-cutting tasks, but agentic workflows augmented with runtime facts significantly improve reliability and speed. An independent SRE benchmark, [OTelBench](https://www.freep.com/press-release/story/145971/quesma-releases-otelbench-independent-benchmark-reveals-frontier-llms-struggle-with-real-world-sre-tasks/), finds frontier models pass only 29% of OpenTelemetry instrumentation tasks across 11 languages, with context propagation as a key failure mode despite much higher scores on coding-only tests. In contrast, Syncause boosted SWE-bench Verified fixes to 83.4% by adding dynamic tracing “Runtime Facts” to the Live-SWE-agent with Gemini 3 Pro, detailing methods and open-sourcing trajectories and code in their [blog](https://syn-cause.com/blog/swe-bench-verified-83) and [repo](https://github.com/Syncause/syncause-swebench). Complementing this, new research on cross-domain workflow generation proposes a decompose–recompose–decide method that surpasses 20-iteration refinement baselines in a single pass, reducing latency and cost for agentic orchestration ([paper](https://arxiv.org/html/2602.11114v1)). For hands-on adoption, the open-source [DeepCode](https://github.com/HKUDS/DeepCode) project provides multi-agent “Text2Backend” capabilities to prototype structured, telemetry-aware coding agents.

calendar_today 2026-02-12
quesma otelbench opentelemetry google-gemini-3-pro syncause

GLM-5 and MiniMax M2.5 push low-cost, agentic coding into production range

Two Chinese releases—Zhipu AI’s GLM-5 and MiniMax M2.5—signal a shift toward affordable, agentic coding models that challenge frontier systems on practical benchmarks. Zhipu AI’s GLM-5 is positioned as an MIT-licensed open model with a native Agent Mode that rivals proprietary leaders on multiple benchmarks, with a deep-dive detailing its pre-launch appearance under a pseudonym and hints from vLLM pull requests ([official overview](https://z.ai/blog/glm-5?_bhlid=d84a093754c9e11cb0d2e9ff416fd99cb5f0e2da), [leak analysis](https://medium.com/reading-sh/glm-5-chinas-745b-parameter-open-source-model-that-leaked-before-it-launched-b2cfbafe99ef?source=rss-8af100df272------2), [weights claim](https://medium.com/ai-software-engineer/glm-5-arrive-with-a-bang-from-vibe-coding-to-agentic-engineering-disrupts-opus-b2b13f02b819)). MiniMax’s M2.5 posts strong results on coding and agentic tasks—80.2% SWE-Bench Verified, 51.3% Multi-SWE-Bench, 76.3% BrowseComp—while running 37% faster than M2.1 and costing roughly $1/hour at 100 tokens/sec (or $0.30/hour at 50 tps), with speed reportedly matching Claude Opus 4.6 ([release details](https://www.minimax.io/news/minimax-m25)). For developer workflows, quick-start videos show GLM-5 (and similarly Kimi K2.5) slotting into Claude Code with minimal setup, lowering trial friction inside existing IDEs ([GLM-5 with Claude Code](https://www.youtube.com/watch?v=Ey-HW-nJBiw&pp=ygURQ3Vyc29yIElERSB1cGRhdGU%3D), [Kimi K2.5 with Claude Code](https://www.youtube.com/watch?v=yZtLwOhmHps&pp=ygURQ3Vyc29yIElERSB1cGRhdGU%3D)).

calendar_today 2026-02-12
zhipu-ai glm-5 minimax minimax-m25 openrouter

Mixture-of-Models router tops single LLMs on SWE-Bench Verified (75.6%)

A lightweight router that clusters tasks and selects the historically best model per cluster hit 75.6% on SWE-Bench Verified, narrowly outperforming top single-model baselines (~74%). Details and methodology are outlined in Nordlys Labs' write-up, including semantic clustering and per-cluster success routing without test-time search or repo execution [Nordlys Labs blog](https://nordlyslabs.com/blog/hypernova)[^1]. The open-source framework implementing this mixture-of-models approach is available here [Nordlys GitHub](https://github.com/Nordlys-Labs/nordlys)[^2]. [^1]: Adds: methodology, routing design, and reported benchmark results. [^2]: Adds: production-ready code for the router and integrations.

calendar_today 2026-02-07
nordlys-labs nordlys swe-bench swe-bench-verified llm-routing

E2E coding agents: 27% pass, cheaper scaling, and safer adoption

A new end-to-end benchmark, [ProjDevBench](https://arxiv.org/html/2602.01655v1)[^1] with [code](https://github.com/zsworld6/projdevbench)[^2], reports only 27.38% acceptance for agent-built repos, highlighting gaps in system design, complexity, and resource management. Efficiency is improving: [SWE-Replay](https://quantumzeitgeist.com/17-4-percent-performance-swe-replay-achieves-gain-efficient/)[^3] recycles prior agent trajectories to cut test-time compute by up to 17.4% while maintaining or slightly improving fix rates. For evaluation and safety, Together AI shows open LLM judges can beat GPT‑5.2 on preference alignment ([post](https://www.together.ai/blog/fine-tuning-open-llm-judges-to-outperform-gpt-5-2at/))[^5], Java teams get a pragmatic path via [ASTRA‑LangChain4j](https://quantumzeitgeist.com/ai-astra-langchain4j-achieves-llm-integration/)[^6], and an open‑weight coding LM targets agentic/local dev ([Qwen3‑Coder‑Next](https://www.youtube.com/watch?v=UwVi2iu-xyA&pp=ygURU1dFLWJlbmNoIHJlc3VsdHM%3D))[^7]. [^1]: Adds: defines an E2E agent benchmark with architecture, correctness, and refinement criteria plus pass-rate findings. [^2]: Adds: benchmark repository for tasks, harnesses, and evaluation assets. [^3]: Adds: test-time scaling via trajectory replay with up to 17.4% cost reduction and small performance gains on SWE-Bench variants. [^4]: Adds: DPO-tuned open "LLM-as-judge" models outperform GPT‑5.2 on RewardBench 2 preference alignment, with code/how-to. [^5]: Adds: security analysis of self-propagating adversarial prompts ("prompt worms") and the OpenClaw agent network example. [^6]: Adds: Java integration pattern for agent+LLM via ASTRA modules and LangChain4J, including BeliefRAG and Maven packaging. [^7]: Adds: open-weight coding model positioned for agentic workflows and local development.

calendar_today 2026-02-03
projdevbench swe-replay swe-bench-verified swe-bench-pro astra