Scope. This digest covers the harness (also called scaffold, agent framework, runtime, or agent-computer interface) that surrounds a language model to make it a coding agent: the control loop, tool surface, context and memory management, verification, orchestration, sandbox and permission model, and cost controls. It covers both the academic literature and primary industry sources (Anthropic, OpenAI, Microsoft/GitHub, Google, Cognition, Cursor, Factory, All Hands, Princeton/Stanford SWE-agent, Augment, Replit, Amazon, Sourcegraph). It does not cover general-purpose agent frameworks except where they are used for coding.
Method. Four channels, 2026-09-10: (1) WebSearch/WebFetch of arXiv abstract pages for harness-design literature; (2) the same for evaluation harnesses and benchmarks; (3) primary industry engineering posts, docs, model cards and repositories; (4) eight Consensus "Deep" searches (consensus.app, Pro) whose cited papers were resolved back to primary pages. Records were written only from pages actually fetched. A separate adversarially-prompted verification agent re-fetched 102 of the records (every low/medium-confidence record, every record whose number is quoted here, plus a sample) and found 87 confirmed, 15 minor metadata fixes, and zero fabrications or missing numbers; the fixes were applied. Final dataset: 268 records (182 academic, 70 industry, 19 open-source; 136 from 2026, 87 from 2025, 39 from 2024). Bracketed tokens like [Noneharness-bench] are record ids in sources.json.
The one-paragraph answer. The harness has become a first-class, independently studied engineering object. Three source-code taxonomies published in 2026 converge on roughly the same decomposition, industry now uses the word "harness" in the titles of its engineering posts, and there is a small but real experimental literature that holds the model fixed and varies the harness (or vice versa). That literature disagrees: the largest crossed study finds a 23.8-point spread across harnesses at fixed model and task [Noneharness-bench]; a longitudinal study of 35 harness releases at fixed model finds no significant change in resolve rate but a doubling of cost [Nonedont-blame-the-llm-scaffolding-evolution]; a 9,374-trajectory study finds the model dominates and the framework gap shrinks each model generation [Nonebeyond-resolution-rates-2026]. The two vendors that train models on their own harness (Cognition, Cursor) and the two that say a harness "encodes assumptions about what the model can't do" (Anthropic, OpenAI) are describing the same phenomenon from opposite ends: harness and model are being co-designed, so their contributions are decreasingly separable from outside.
1. What a harness is, and how the field now decomposes it
The term entered the SE literature through SWE-agent's agent-computer interface (ACI), which reframed the LLM as a new kind of end user needing purpose-built commands and feedback formats [Noneswe-agent-aci-2024]. By 2026 the object of study had widened from the interface to the whole runtime, and three independent source-code studies decomposed it in compatible ways:
- Inside the Scaffold analysed 13 open-source coding agents along 12 dimensions in three layers — control architecture, tool/environment interface, resource management — and found five composable loop primitives (ReAct, generate-test-repair, plan-execute, multi-attempt retry, tree search), with 11 of 13 agents composing several; tool counts ranged from 0 to 37; and seven distinct context-compaction strategies were in use [Noneinside-the-scaffold-2026].
- Code as Agent Harness organises the space as harness interfaces / harness mechanisms / scaling the harness, and distinguishes four planning types by locus of control: linear decomposition, structure-grounded, search-based, orchestration-based [Nonecode-as-agent-harness-survey-2026].
- From Question Answering to Task Completion names six coupled runtime responsibilities — observation, context, control, action, state, verification — and traces a paradigm progression from prompt optimisation to context engineering to harness engineering to agent-native training with model–harness co-evolution [Noneqa-to-task-completion-harness-survey-2026].
Two further framings matter for measurement. AI Harness Engineering formalises eleven component responsibilities (context selection, tool access, project memory, failure attribution, verification, intervention recording, …) and a ladder of harness levels from "final patch only" up to reproduction logs, failure attributions and structured verification reports [Noneai-harness-engineering-runtime-substrate-2026]. Harness Engineering: Eleven Systems dissects roughly four million lines of code across Claude Code, Codex CLI, Gemini CLI, OpenHands, Aider, mini-SWE-agent and others [Noneharness-engineering-eleven-systems-2026]; a companion exploratory study looks at how practitioners configure these tools in the wild [Noneharness-engineering-configuration-2026], and a design-space paper takes Claude Code specifically as its object [Nonedive-into-claude-code-2026].
Older SE surveys still use a perception/memory/action framing [Noneliu-agent4se-survey-2024] [Nonewang-agents-in-se-survey-2025] [Nonehe-llm-mas-se-2024], which does not capture verification, permissions or cost — the three dimensions where the 2026 literature is most active. The roadmap papers [Noneagentic-se-pillars-roadmap-2025] [Noneprogramming-with-trust-2025] [Nonedeterminism-to-delegation-2026] and the challenges paper [Nonellm-agentic-systems-se-challenges-2026] are the natural bridge.
Working taxonomy used in this digest (collapsed from the three studies above): (1) action representation and tool interface; (2) control loop; (3) context and memory; (4) verification and feedback; (5) multi-agent orchestration; (6) sandbox and permissions; (7) cost control; (8) human interaction.
2. The design space, dimension by dimension
2.1 Action representation and tool interface
The three live options are bash-only, structured tools (JSON-schema function calls), and executable code as the action language [Nonecodeact-2024]. The best-controlled comparison is The Devil Is in the Interface: 11,700 trajectories, six tool designs, three models. Structured low-level interfaces gave 4.7× better run-to-run consistency than bash-only, and CodeAct-style Python interfaces matched accuracy with 41.6% fewer steps and 56.3% fewer tokens [Nonedevil-in-the-interface-2026]. A regime × agent ablation found that restricting an agent to execute_code changed cost but not pass rate, and the optimum depended jointly on task regime and agent design [Nonerestricting-execute-code-ablation-2026]. The Bitter Lesson of Tool Calling argues the trend runs toward fewer, more general tools [Nonebitter-lesson-tool-calling-2026], which is what industry has done: Anthropic's 2025 SWE-bench Verified scaffold was bash plus one string-replace edit tool [Noneanthropic-swe-bench-sonnet-2025], and mini-SWE-agent reaches >74% on SWE-bench Verified with ~100 lines of Python and bash alone [Nonemini-swe-agent-repo].
Tool description quality is its own variable: descriptions written for humans tolerate ambiguity agents cannot resolve [Noneeasytool-2024]; Trace-Free+ cuts accuracy degradation by 29.23% as catalogs grow past 150 tools [Nonetrace-free-plus-tool-descriptions-2026]; schema-first contracts reduce syntactic misuse but not semantic action errors [Noneschema-first-tool-apis-2026]. Anthropic's own guidance on writing tools for agents [Noneanthropic-writing-tools-for-agents] and the MCP-over-code-execution pattern (150k → 2k tokens, a 98.7% saving, in Anthropic's example) [Noneanthropic-code-execution-with-mcp] [Nonemcp-orchestration-to-code-execution-2026] are the industry counterparts. MCP itself is now a Linux-Foundation-hosted spec (revision 2026-07-28, with Tasks and Skills extensions) [Nonemcp-specification-2026-07-28], with empirical studies of its applications and security surface [Nonemcp-applications-empirical-2026] [Nonemcp-security-maintainability-2025].
2.2 Control loop
The dominant production pattern is a single ReAct-style loop over a growing context. Its structural weaknesses — implicit step dependencies, unbounded recovery loops, mutable history — motivate the Structured Graph Harness (control flow lifted into an explicit DAG with immutable plan versioning) [Nonestructured-graph-harness-2026] and LLM-as-Code / agentic programming, where deterministic code owns control flow and the LLM is called only for reasoning, bounding context by call depth rather than step count [Nonellm-as-code-agentic-programming-2026]. Search-based loops (MCTS over trajectories) [Noneswe-search-mcts-2024], test-time scaling with multiple attempts [Nonetrae-agent-2025], and self-evolving scaffolds where the agent builds its own tools at runtime (Live-SWE-agent, 77.4% on SWE-bench Verified) [Nonelive-swe-agent-2025] sit at the other end. Industry's 2026 move is dynamic workflows: the model generates the harness per task [Noneanthropic-dynamic-workflows-harness], and Anthropic's long-running-app post argues that "every component in a harness encodes an assumption about what the model can't do on its own" and reports that Opus 4.6 needed less scaffolding than 4.5 [Noneanthropic-harness-design-long-running-apps]. The Agentless result — 32.00% on SWE-bench Lite at $0.70 with a fixed three-phase pipeline, beating agentic scaffolds at the time — remains the canonical argument that loops are not free [Noneagentless-2024].
2.3 Context and memory
This is where the 2025–2026 literature grew fastest (nineteen records came from a single Consensus query). The shared diagnosis: append-only histories produce redundancy, semantic drift and reasoning collapse well before the window limit [Nonecontext-as-a-tool-cat-2025] [Noneactive-context-compression-focus-2026]; production systems (Claude Code, Codex, LangChain, LlamaIndex) compact by LLM summarisation [Noneparallel-context-compaction-2026] [Noneanthropic-claude-code-context-window-docs]. Summarisation is 90–99% token reduction and always lossy, its salience judgement need not match the downstream agent's needs, and synchronous compaction stalls inference for tens of seconds [Noneparallel-context-compaction-2026] [Nonecwl-structured-context-eviction-2026]. The safety consequence is now measured: Governance Decay shows in-context policy violations rising from 0% (policy visible) to 30% after compaction, 59% for some models, across 1,323 episodes; constraint pinning restores 0% [Nonegovernance-decay-2026].
Responses fall into three families. Learned, agent-initiated compaction: Context-as-a-Tool / SWE-Compressor (57.6% SWE-bench Verified) [Nonecontext-as-a-tool-cat-2025], SWE-MeM (60.2% with a 30B model) [Noneswe-mem-2026], CompactionRL (+7.0 points for GLM-4.5-Air to 66.8%) [Nonecompactionrl-2026], Context-Folding (10× smaller active context) [Nonecontext-folding-2025], AgentFold [Noneagentfold-2025], ACON [Noneacon-2025], MEM1 [Nonemem1-2025], Memory-as-Action [Nonememory-as-action-2025]. Lossless or externalised memory: Memex indexed archives [Nonememex-rl-2026], ACM's manage_context/query_memory tools [Noneacm-agentic-context-management-2026], dependency-graph eviction instead of summarisation [Nonecwl-structured-context-eviction-2026], Git-Context-Controller's COMMIT/BRANCH/MERGE (>80% on SWE-bench Verified) [Nonegit-context-controller-2025], SimpleMem [Nonesimplemem-2026], programmatic context environments [Nonecontext-as-environment-scroll-2026], and the observation that coding agents beat dedicated long-context methods by 17.3% simply by using the file system [Nonecoding-agents-long-context-processors-2026]. Validation and external management: Slipstream runs the compactor asynchronously and judges the summary against the agent's continued reasoning (+8.8 pp, −39.7% latency) [Noneslipstream-compaction-validation-2026]; AdaCoM trains an external manager for a frozen agent and finds a fidelity–reliability trade-off [Noneadacom-2026]; a rate–distortion view unifies KV-cache eviction and long-term memory and notes that repeated compaction is almost never measured [Nonerate-distortion-memory-compaction-2026].
Industry converged in 2025 on a triad — compaction, structured note-taking to files, and sub-agent context isolation [Noneanthropic-effective-context-engineering] [Noneanthropic-effective-harnesses-long-running] — plus project memory files (CLAUDE.md, AGENTS.md, auto-memory) [Noneanthropic-claude-code-memory-docs] [Noneagents-md-convention], Aider's repository map [Noneaider-repo-map], and progressive disclosure via Agent Skills [Noneanthropic-agent-skills] [Noneagentskills-io-standard]. Cross-session repository memory is nearly unstudied academically; [Nonerepository-memory-localization-2025] is the exception.
2.4 Verification and feedback
The universal motivation is that agents grade their own work too generously. Academic results: separating the test agent from the repair agent is worth +11.4 pp on SWE-bench Verified with no stronger model (ExecCritic) [Noneexec-critic-2026]; SWE-Gym's verifier layer produced 32.0% Verified from a fine-tuned open model [Noneswe-gym]; adversarial review structures disagreement [Noneadversarial-review-2026]; Looping Is Not Reliability argues for state-bound evidence and typed revision contracts [Nonelooping-is-not-reliability-2026]; a five-level verification ladder finds 70% of analysed loops in the autonomous zone [Nonestop-handholding-loops-2026]; process-level verification in multi-agent systems is high-variance and often marginal, with trained judges beating general LLMs [Nonemas-prove-2026]; SagaLLM adds transactional validation [Nonesagallm-2025]. Industry counterparts: Anthropic's planner/generator/evaluator split [Noneanthropic-harness-design-long-running-apps], Replit's isolated testing subagent ($0.20 median test cost, autonomy 20 → 200+ minutes) [Nonereplit-agent3-self-testing], Cognition's Devin Review (~2 bugs per PR, 58% severe) [Nonecognition-devin-2], and Antigravity's Artifacts as verification surface [Nonegoogle-antigravity-launch]. Trajectory-level studies show failures are epistemic, start early and stay hidden [Nonefailure-as-a-process-2026] [Noneunderstanding-se-agents-trajectories-2025] [Nonecode-agent-behaviour-trajectories-2025].
2.5 Multi-agent orchestration
The evidence is mixed and the field reversed itself in 2026. Gains on hard, long tasks: AgentForge +26–28 points over single-agent baselines on SWE-bench Lite via execution-grounded verification [Noneagentforge-2026]; Agyn 72.4% on SWE-bench 500 [Noneagyn-2026]; TheBotCompany better on the hardest ProjDevBench tasks but less efficient on easy ones [Nonethebotcompany-2026]; Anthropic's research system +90.2% over single Opus 4, with token usage explaining 80% of variance [Noneanthropic-multi-agent-research-system]; a C compiler built by parallel Claudes [Noneanthropic-c-compiler-parallel-claudes]. Diminishing returns: a 260-configuration study in Nature Machine Intelligence finds a capability-saturation threshold beyond which more agents stop helping, predicting the multi-agent effect in 94% of SWE-bench Verified validation configurations [Nonecapable-lms-outgrow-collaboration-2026]; a single-agent README pipeline matches multi-agent quality with 86% fewer tokens [Noneillusion-agentic-complexity-readme-2026]; waterfall-style role pipelines cut functional correctness by 37.8% (GPT-4o-mini) and 39.8% (DeepSeek-Chat) while helping Claude-3.5-Haiku [Nonesoftware-process-models-multiagent-2025]; hybrid cascading gets +1.1–12% accuracy at −20% cost [Nonesingle-or-multi-agent-both-2025]. Cognition's Don't Build Multi-Agents [Nonecognition-dont-build-multi-agents] was revised in April 2026 to Multi-Agents: What's Actually Working [Nonecognition-multi-agents-working]; the position now shared with Anthropic's read-only Explore/Plan subagents [Noneanthropic-claude-code-subagents-docs] is writes single-threaded, reads and reasoning parallel. Protocol-level fixes (SEMAP, −69.6% failures) [Nonesemap-protocol-driven-2025], verification-aware planning [Noneverimap-2025] [Nonevmao-verified-multiagent-orchestration-2026], and Microsoft's ledger-based Magentic-One [Nonemicrosoft-magentic-one] / Agent Framework [Nonemicrosoft-agent-framework] are the orchestration mechanisms on offer; the foundational systems remain MetaGPT, ChatDev, AgentCoder, MapCoder, CodeR, HyperAgent [Nonemetagpt-2023] [Nonechatdev-2023] [Noneagentcoder-2023] [Nonemapcoder-2024] [Nonecoder-task-graphs-2024] [Nonehyperagent-2024].
2.6 Sandbox and permissions
Named in every taxonomy, measured in almost none — a survey of 39 execution-security papers found isolation architectures and capability models are never compared on a shared benchmark, and policy enforcement fails 69–98% of the time against real denylists [Nonemcp-execution-security-balkanization-2026]. Frontier models can find and exploit sandbox escapes when misconfigurations exist [Nonesandbox-escape-bench-2026]. Academic mechanisms: transactional sandboxes (100% interception and rollback, 14.5% overhead) [Nonefault-tolerant-sandboxing-2025], AgentSpec runtime DSL (>90% unsafe executions prevented) [Noneagentspec-2025], capability-labelled MCP [Noneverifiably-safe-tool-use-2026], classical security principles applied to agents [Nonellm-agents-security-principles-2025], constraint substrates that raise backdoor-detection recall from 54.5% to 90.9% with a small reviewer [Nonesteerability-via-constraints-2026], LlamaFirewall [Nonellamafirewall-2025], human-takeover sandboxes [Noneagentbay-2025], and a taxonomy of 547 confirmed operational safety failures (>65% during bug fixing and setup) [Nonewhat-breaks-when-llms-code-2026]. Industry discloses this layer in unusual detail: bubblewrap/Seatbelt plus egress proxy cutting permission prompts 84% [Noneanthropic-claude-code-sandboxing], gVisor and sealed VMs [Noneanthropic-how-we-contain-claude], a transcript classifier replacing human approval in auto mode [Noneanthropic-claude-code-auto-mode], Codex's read-only / workspace-write / full-access modes with network off by default [Noneopenai-codex-config-sandbox], and GitHub's zero-secret chroot jail with vetted "safe outputs" [Nonegithub-agentic-workflows-security].
2.7 Cost control
Cost is harness-determined to a degree accuracy is not. The Scaffold Effect finds up to 40× difference in tokens per solved task across harnesses but only 0–8 pp within-model pass-rate spread [Nonescaffold-effect-coding-agents-2026]; SWE-Effi finds model × scaffold interaction dominates, with failures burning 4× the resources of successes [Noneswe-effi]; AI Agents That Matter made cost a first-class axis [Noneai-agents-that-matter]; HAL reports cost across 21,730 rollouts [Nonehal-holistic-agent-leaderboard]; token-consumption prediction [Nonetoken-consumption-agentic-coding-2026] and even language choice ("tokenmaxxing") [Nonetokenmaxxing-languages-2026] are now studied. Industry's answer is the same triad as context management plus effort controls: Opus 4.5 reported +4.3 pp SWE-bench at 48% fewer tokens [Noneanthropic-claude-opus-4-5-eval-harness]; Codex's retained reasoning and compaction moved GPT-5.6 on ARC-AGI-3 from 13.3% to 38.3% at one-sixth the output tokens — a harness change, same model [Noneopenai-codex-as-a-platform].
2.8 Human interaction
The position paper Humans are Missing from AI Coding Agent Research finds essentially no harness-level metrics for interaction quality [Nonehumans-missing-coding-agent-research-2026]; HULA [Nonehula-human-in-the-loop-2024], ToM-SWE user modelling [Nonetom-swe-2025], and the copilot-vs-agent user study [Nonecode-with-me-or-for-me-2025] are the exceptions. The field studies in §5 are the empirical substitute.
3. Industry landscape
| Org | Product | Harness disclosed? | Model co-designed with harness? | Sandbox | Orchestration | Records |
|---|---|---|---|---|---|---|
| Anthropic | Claude Code / Agent SDK / Managed Agents | Yes — docs, engineering posts, SDK ships the harness | Yes | bubblewrap/Seatbelt + egress proxy; gVisor elsewhere | Subagents with isolated context, worktrees, dynamic workflows | [Noneanthropic-claude-code-docs-overview] [Noneanthropic-agent-sdk-overview] [Noneanthropic-managed-agents] |
| OpenAI | Codex CLI / cloud / App Server | Yes — harness open-sourced Aug 2026 | Yes (codex-1, GPT-5-Codex, GPT-5.3-Codex) | read-only / workspace-write / full; network off by default | Threads/Turns/Items over JSON-RPC | [Noneopenai-codex-as-a-platform] [Noneopenai-codex-app-server] [Noneopenai-harness-engineering] [Noneopenai-gpt-5-3-codex] |
| GitHub / Microsoft | Copilot cloud agent, Agent HQ, Agent Framework | Partial | No (multi-vendor) | Ephemeral Actions env, chroot jail, zero-secret | Mission Control across vendors; Magentic ledgers | [Nonegithub-copilot-cloud-agent-docs] [Nonegithub-agent-hq] [Nonemicrosoft-agent-framework] |
| Antigravity (CLI), ADK, Gemini CLI (sunset) | Partial | No | not disclosed | Agent Manager, artifacts | [Nonegoogle-antigravity-launch] [Nonegoogle-gemini-cli-to-antigravity-cli] [Nonegoogle-adk] | |
| Cognition | Devin, SWE-1.5/1.7 | Partial | Yes — RL on the shipping harness | VM hypervisor | Single-threaded writes + advisory agents | [Nonecognition-swe-1-5] [Nonecognition-devin-2] [Nonecognition-multi-agents-working] |
| Cursor | Composer | Partial | Yes — RL with production tools | 100k+ cloud sandboxes in training | in-editor agents | [Nonecursor-composer-rl] |
| Factory | Droid | Partial (principles) | No — model-agnostic | not disclosed | single agent | [Nonefactory-droid-terminal-bench] |
| All Hands | OpenHands SDK | Yes — paper + source | No | sandboxed, local↔remote | multi-LLM routing | [Noneopenhands-2024] [Noneopenhands-agent-sdk-2025] |
| Princeton/Stanford | SWE-agent, mini-SWE-agent, SWE-ReX | Yes — fully | No | subprocess/container | none | [Noneswe-agent-aci-2024] [Nonemini-swe-agent-repo] [Noneswe-rex] |
| Augment, Replit, Amazon, Sourcegraph, Aider, Cline | various | Yes / partial | No | various | various | [Noneaugment-swebench-agent] [Nonereplit-agent3-self-testing] [Nonekiro-docs] [Nonesourcegraph-amp-manual] [Noneaider-edit-formats] [Nonecline-repo] |
Recurring patterns: one engine, many surfaces (terminal, IDE, web, CI share a core); externalised state (progress files, git, wikis) as the answer to context limits; verification as a separate agent; writes single-threaded, reads parallel; tool interfaces collapsing into code interfaces; progressive disclosure everywhere; permissions moving from human gates to classifiers plus OS enforcement; and cross-vendor standardisation on MCP, AGENTS.md, Agent Skills and A2A [Nonea2a-protocol].
What industry discloses: architecture narratives, sandbox mechanisms in detail, context-management technique, and — increasingly — full harness source. What it withholds: production system prompts, the exact scaffold behind headline benchmark numbers (Anthropic's 2025 SWE-bench post is the outlier full disclosure), RL-environment composition for co-designed models, model routing decisions, and evaluation infrastructure configuration — which Anthropic has itself shown moves Terminal-Bench 2.0 by 6 pp (p<0.01) from sandbox RAM/CPU alone, larger than typical inter-model gaps [Noneanthropic-infrastructure-noise-evals].
4. Evaluation harnesses and their validity
Three grains. Sub-task benchmarks (HumanEval, MBPP, LiveCodeBench) have essentially no harness, so their variance is model variance [Nonehumaneval-codex] [Nonembpp] [Nonelivecodebench]. Repository-level, execution-graded benchmarks are the centre: SWE-bench and Verified/Lite/Multimodal/Multilingual/Pro [Noneswe-bench-original] [Noneswe-bench-verified-openai] [Noneswe-bench-multimodal] [Noneswe-bench-pro], the derivative families (SWE-PolyBench, Multi-SWE-bench, SWE-bench-Live, SWE-rebench, SWE-Bench++, SWE-EVO, SWE-Compass, SWE Atlas, SWE-Bench Mobile) [Noneswe-polybench] [Nonemulti-swe-bench] [Noneswe-bench-live] [Noneswe-rebench] [Noneswe-bench-plusplus] [Noneswe-evo] [Noneswe-compass-2025] [Noneswe-atlas-2026] [Noneswe-bench-mobile-2026], and training environments that reuse the contract (SWE-Gym, SWE-smith) [Noneswe-gym] [Noneswe-smith-2025]. Environment benchmarks — Terminal-Bench 2.0 and Harbor, TerminalWorld, OSWorld, AppWorld, τ-bench — ship an environment and invite any containerisable agent [Noneterminal-bench-2] [Noneharbor-terminal-bench-announcement] [Noneterminalworld] [Noneosworld] [Noneappworld] [Nonetau-bench].
How each treats the harness. SWE-bench separates harness (Docker, patch grading) from agent [Noneswe-bench-harness-repo] and now runs two leaderboards: a default "bash only" board that puts every model in the same mini-SWE-agent, and a full board for arbitrary systems [Noneswebench-bash-only-leaderboard] [Noneswebench-leaderboard-variants]. Epoch AI, DeepSWE and SWE-MERA independently adopted the one-fixed-scaffold design [Noneepoch-swe-bench-verified] [Nonedeepswe-datacurve] [Noneswe-mera]. Terminal-Bench is the only leaderboard that structurally separates a "model" column from an "agent" column [Noneterminal-bench]. HAL and Inspect provide the infrastructure [Nonehal-holistic-agent-leaderboard] [Nonehal-insights-page] [Noneinspect-ai]; a leaderboard dissection profiles who submits what [Nonedissecting-swe-bench-leaderboards].
Validity threats are quantified. SWE-Bench+ found 32.67% solution leakage and 31.08% weak tests, dropping SWE-agent+GPT-4 from 12.47% to 3.97% [Noneswe-bench-plus]; OpenAI's Verified build discarded 68.3% of instances and dataset cleaning alone doubled GPT-4o's score [Noneswe-bench-verified-openai]; in February 2026 OpenAI retired Verified after finding 59.4% of audited hard failures were test flaws [Noneopenai-swe-bench-verified-retired]; UTBoost found 345 wrongly-passed patches and changed 40.9% of Lite rankings [Noneutboost-2025]; The SWE-Bench Illusion measured memorisation (76% vs 53% file-path accuracy from issue text alone) [Noneswe-bench-illusion], corroborated by a 3× localisation gap against off-benchmark repos [Noneswe-bench-verified-memory-2025]; benchmark mutation costs 20–40% relative [Nonesaving-swe-bench-mutation]; TerminalWorld correlates with Terminal-Bench at only r = 0.20 [Noneterminalworld]. Single-run reporting is unsound: capability and reliability rankings invert at long horizons [Nonebeyond-pass-at-1-reliability] [Nonetowards-science-agent-reliability], and FP8 serving diverges from control forks on 90–96% of temperature-0 states [Nonereplay-gap]. Contamination is now measurable [Nonecross-context-verification-2026] [Nonequantifying-contamination-code-2024] [Nonedycodeeval-2025] but never used as a covariate when attributing score deltas. Only ~39% of 178 surveyed benchmarks look beyond implementation [Nonesdlc-benchmark-survey-2025]. The position paper Coding Benchmarks Are Misaligned with Agentic SE makes the structural point: benchmarks treat an agent as a model, not a composite system, and give no per-component signal [Noneposition-coding-benchmarks-misaligned-2026].
5. Field evidence: agents in real repositories
The AIDev dataset (932,791 agent-authored PRs, 116,211 repositories, 72,189 developers, five agents) [Noneaidev-dataset-2026] underpins a cluster of MSR-style studies. Adoption reached 22.2–28.7% of GitHub projects by early 2026 [Noneagentic-much-adoption-github-2026]. Claude Code PRs merged at 83.8% vs 91.0% for humans, 54.9% unmodified [Noneagentic-coding-prs-watanabe-2025]; across 7,156 PRs task type dominates (documentation 82.1% vs features 66.1%) and no agent wins everywhere [Nonecomparing-ai-coding-agents-pinna-2026]; only 35.7% of rejections are clear agent failures [Nonewhy-agentic-prs-merged-rejected-2026]. Velocity gains are front-loaded and vanish in repositories with prior AI-IDE use, while static-analysis warnings rise ~18% and cognitive complexity ~39% [Noneai-ides-or-autonomous-agents-2026]; agent code churns more and is retained less [Noneautonomous-agent-contributions-wild-2026] [Nonehow-do-agents-contribute-2026] [Nonehow-agents-modify-code-2026]; 26.1% of agent commits are refactorings, mostly low-level [Noneagentic-refactoring-2025]; security PRs are ~4% and merge slower [Nonesecurity-age-ai-teammates-2026]; core and peripheral developers use agents differently [Nonecore-peripheral-agent-use-2026]; failed PRs are larger and fail CI [Nonewhere-agents-fail-2026] [Nonefix-related-prs-unmerged-2026]; code-review agents as sole gatekeepers merge 23 pp below human review [Nonecode-review-agents-empirical-2026]; agents partition work along a collaborator–assistant spectrum with merge governance staying human [Nonecollaborator-or-assistant-2026] [Nonehow-agents-communicate-prs-2026]. Controlled studies: Copilot raised PRs per week 12.92–21.83% at Microsoft in a 1,974-developer field experiment [Nonecopilot-productivity-field-experiment-2024]; AI assistance doubled task completeness but cut developers' ability to answer questions about their own code by 12.5% [Nonemore-code-less-understanding-2026]; METR's time-horizon measure is the long-task counterpart [Nonemetr-time-horizons-paper].
For the harness question these studies matter because they are the only place where products (harness + model, undisclosed) are compared on the same task distribution in the wild — and the spread between products is of the same order as the spread between task types.
6. Model versus harness: what the attribution evidence actually says
Harness-varied, model-fixed. - Harness-Bench: 6 harnesses × 8 models, 106 tasks, 5,194 trajectories. 23.8 pp spread (52.4% → 76.2%) at identical model and task; stronger models less harness-sensitive; recommends reporting per model–harness pair [Noneharness-bench]. - Scaffold Effects on GAIA: scaffold alone moves accuracy up to 28 pp within one model, and the strongest model gained most [Nonescaffold-effects-gaia-2026] — the opposite direction from Harness-Bench. - Don't Blame the LLM: model pinned, 35 sequential Qwen Code CLI releases, 50 SWE-bench Verified tasks. No significant change in resolve rate (mean 30.5%, range 23–39%) but ~2× tokens and tool calls; practitioners misattribute the swings to the model [Nonedont-blame-the-llm-scaffolding-evolution]. - The Scaffold Effect: up to 40× tokens per solved task across harnesses, 0–8 pp pass-rate spread; failure fingerprints replicate across models within a harness [Nonescaffold-effect-coding-agents-2026]. - SWE-Bench Mobile: up to 6× same-model spread across agent frameworks [Noneswe-bench-mobile-2026]. Agent-framework evaluation on code-centric tasks [Noneagent-frameworks-empirical-eval-2025]. - Same-model inversions in industry: Droid with Sonnet beat every other agent running Opus on Terminal-Bench (58.8%) — "agent design, not just choice of model, is the decisive factor" [Nonefactory-droid-terminal-bench]; Codex's harness change moved a fixed model from 13.3% to 38.3% [Noneopenai-codex-as-a-platform].
Model-varied, harness-fixed or model-dominant. - Beyond Resolution Rates: 9,374 trajectories, 19 agents, 8 frameworks, 14 LLMs. Agents sharing an LLM agree on far more tasks than agents sharing a framework; the framework gap shrinks each model generation [Nonebeyond-resolution-rates-2026]. - Capability saturation: 260 configurations; above a threshold, coordination stops helping; single-agent baseline is the best predictor [Nonecapable-lms-outgrow-collaboration-2026]. - Augment: "scores on SWE-bench Verified are largely driven by the quality of the foundation model"; prompt optimisation plateaued, ensembling added 3–8% at unusable cost [Noneaugment-swebench-agent]. - Plain CLI agents matched some published specialised-harness scores; the recommendation is model-matched plain-agent baselines before crediting architecture [Nonebaselines-before-architecture-2026]. - mini-SWE-agent's >74% from ~100 lines is the strongest single datapoint that the model carries the load once it is good enough [Nonemini-swe-agent-repo].
Interaction dominates. SWE-Effi: the same scaffold scores 28% with one model and 10% with another at >18× the token cost [Noneswe-effi]; waterfall pipelines help one model and hurt two others [Nonesoftware-process-models-multiagent-2025]; procedural fingerprints identify the agent from trace shape with 85.7% accuracy, i.e. harnesses impose behaviour independent of the model [Noneagent-trajectories-as-programs-2026]; infrastructure alone moves scores by several points [Noneanthropic-infrastructure-noise-evals].
Reading the disagreement. The studies are not measuring the same thing. Harness-Bench and the GAIA study vary rich harnesses on general agent tasks; The Scaffold Effect and Don't Blame the LLM vary coding harnesses on SWE-bench and find the accuracy effect small and the cost effect large; Beyond Resolution Rates looks at leaderboard submissions where harness and model were chosen together. All are small (50–106 tasks, k ≤ 3, 2–8 harnesses) relative to the run-to-run variance the reliability papers document. Nobody has yet published a variance decomposition (model / harness / interaction / run / infrastructure) with confidence intervals on a repository-level SE benchmark.
The structural point. Two vendors train models on their own shipping harness [Nonecognition-swe-1-5] [Nonecursor-composer-rl]; SWE-smith trains open models on one scaffold's trajectories and no paper tests transfer to another harness [Noneswe-smith-2025]; Anthropic and OpenAI both describe the harness as a decaying encoding of model deficits [Noneanthropic-harness-design-long-running-apps] [Noneopenai-harness-engineering]; and the survey literature names "agent-native training with model–harness co-evolution" as the current paradigm [Noneqa-to-task-completion-harness-survey-2026]. When the training distribution is the harness, an external experiment that swaps one for the other is no longer sampling from the population the product was built for. That is the sense in which model-versus-harness attribution at the frontier is structurally, not merely practically, unanswerable — which is the thesis of the soa-agentic-se paper, and this digest finds the field's own evidence pointing the same way.
7. Gaps
- No variance decomposition with intervals on a repository-level benchmark; attribution studies are underpowered against documented run-to-run variance.
- Fixed-scaffold leaderboards buy comparability with a minimal bash loop no product resembles; the external-validity gap is unquantified.
- Cross-session and project memory is nearly unstudied; sandbox and permission design is named everywhere and compared nowhere.
- Training entangles the variables (scaffold-trained models, co-designed products), and no study tests transfer of a scaffold-trained model to another harness.
- Contamination and infrastructure are measurable confounds that are never used as covariates.
- Human-interaction quality has no harness-level metrics.
- Repeated compaction — the thing agents actually do — is almost never measured, and its safety effect (governance decay) has one paper.
8. Notes for soa-agentic-se
- The three 2026 taxonomies [Noneinside-the-scaffold-2026] [Nonecode-as-agent-harness-survey-2026] [Noneqa-to-task-completion-harness-survey-2026] and the eleven-responsibility ladder [Noneai-harness-engineering-runtime-substrate-2026] are the natural coding frame for the paper's taxonomy dimension "scaffold"; they postdate the frozen protocol and should be checked against it.
- §6 is the evidence map for the central claim; Harness-Bench, Beyond Resolution Rates, Don't Blame the LLM and The Scaffold Effect are the four papers a reviewer will expect to see reconciled.
- The pilot's five-factor design should consider infrastructure (RAM/CPU) as a nuisance factor given [Noneanthropic-infrastructure-noise-evals].
- SWE-bench Verified's retirement [Noneopenai-swe-bench-verified-retired] and the bash-only board [Noneswebench-bash-only-leaderboard] are both after the known-item sweep and before the 2026-09-30 freeze.