Consensus Deep Searches

The distinction between coding agent scaffold (harness) and underlying model capability is a central tension in SE agent research, with growing evidence that both contribute substantially but are routinely conflated in evaluation and attribution (Sghaier 2026; Gorinova 2026).

Harness as determinant. The harness shapes system prompts, tool access, context management and the iterative loop (Sghaier 2026). First controlled longitudinal study: 35 sequential releases of a coding-agent CLI with LLM held constant, tracing quality fluctuations to specific harness components (Sghaier 2026). Confucius Code Agent: orchestration + memory structures can outperform stronger models (Wong 2025). Zhong 2026 ("AI harness engineering: a runtime substrate") formalizes eleven harness component responsibilities (context selection, tool access, project memory, failure attribution, verification, intervention recording…) and harness "levels" from final-patch-only up to reproduction logs + failure attributions + verification reports.

Misattribution problem. End-to-end scores collapse model, harness and environment into one number; no per-component signal; practitioners blame the model for harness regressions; studies look at outputs (PRs) not harness evolution (Gorinova 2026; Sghaier 2026). Each harness component can shift benchmark score by margins comparable to adjacent model generations (Gorinova 2026).

Contested relative weight. Mehtiyev 2026 ("Beyond resolution rates"): 9,374 trajectories from 19 agents — the LLM is the primary driver of outcome and behaviour; agents sharing an LLM agree on more tasks than agents sharing a framework; framework gap shrinks with each LLM generation. Conversely Sghaier 2026 and Wong 2025 show harness-only effects. Oderinwale 2026 ("Agent trajectories as programs"): scaffold and model leave distinct procedural fingerprints (Moatless → lots of testing; Agentless → lots of browsing); harnesses show less task diversity than models. Kumar 2026 (AgentForge): verified execution feedback and pipeline design matter more than raw model scale.

Implications. Benchmarks treat an agent as a model rather than a composite system (Gorinova 2026). Question should shift to whether the model–harness–environment system produces a verifiably correct, attributed, maintainable change (Zhong 2026). Trace-based evaluation / auditable episode packages (Zhong) and procedural fingerprinting (Oderinwale) are proposed routes to component attribution.

ACI design — commands, feedback formats, tool specs — significantly shapes SE-agent performance; interface quality argued to matter as much as model size (Yang 2024; Bhati 2026; Sigdel 2026). Reframes the LLM agent as a new category of end user needing purpose-built interfaces (Yang 2024).

Core ACI architectures. SWE-agent introduced ACI (file edit, repo nav, test exec), 12.5% pass@1 on SWE-bench at publication (Yang 2024). Bhati 2026 reference architecture places ACI at layer L2 between reasoning scaffold and tools/environment. Variants: Devin (editor/terminal/web), Aider (repo map from static + call-graph analysis, diff patches), AutoCodeRover (code-search APIs), Moatless (search tools + LLM queries), SpecRover (spec-guided, function summaries), HyperAgent (Planner/Navigator/Code Editor/Executor) (Xia 2024/2025; Phan 2024).

Tool description quality. Docs written for humans tolerate ambiguity agents cannot resolve, worse as catalogs grow (Guo 2026; Yuan 2024). EasyTool unifies/concisifies tool docs (Yuan 2024). Trace-Free+ curriculum reduces accuracy degradation by 29.23% at 150+ tools (Guo 2026). Schema-first JSON-Schema tool contracts improve adherence and reduce misuse but not semantic action errors; in a constrained-local-inference pilot, end-task success was zero across all conditions (Sigdel 2026).

Orchestration/trajectories. Multi-tool orchestration over long horizons is the central challenge (Xu 2026). Trajectory motifs/anti-patterns distinguish success from failure (Bouzenia & Pradel 2025). ToolMaker turns papers-with-code into tools, 80% correct (Wölflein 2025).

Simplicity vs complexity. Agentless: 32.00% on SWE-bench Lite with a three-phase localize/repair/validate pipeline — verbose tool setups may exacerbate the human–LLM ability gap (Xia 2024/2025).

Open challenges. Spec quality, retrieval context, planning; ReAct drift; memory/orchestration/verification/benchmark design for long horizons (Xu 2026); alternative modalities (Liu 2024); version control/package management inside agent systems (Wang 2024).

SWE-bench and variants suffer solution leakage, weak tests and contamination that inflate scores and undermine reproducibility (Aleithan 2024; Liang 2025); audits distort rankings and conflate memorization with problem-solving (Liang 2025; Yu 2025; Prathifkumar 2025).

Contamination/memorization. >94% of SWE-bench issues predate common cutoffs (Aleithan 2024). File-path identification from issue text alone: 76% on SWE-bench vs 53% external repos (Liang 2025, "SWE-Bench Illusion"). Claude models 3× better at file localization on Verified than on BeetleBox / SWE-rebench (Prathifkumar 2025). 35% vs 18% consecutive 5-gram overlap (Liang 2025). Riddell 2024 quantifies overlap of code benchmarks with open training corpora.

Leakage / weak tests (table). Solution leakage 32.67% of successful patches; weak tests 31.08% (Aleithan 2024). Combined leakage in commonly-resolved issues 60.83% across three top agents (Xue 2026, SWE-bench+ enhanced). UTBoost: 36 instances with insufficient tests, 345 erroneously passed patches; 40.9% of Lite rankings change, 24.4% of Verified entries (Yu 2025). OpenAI audit: 59.4% of 138 o3 failures caused by test flaws → recommend discontinuing Verified reporting; git-history leakage of gold patches (Song 2026). Filtering drops SWE-Agent+GPT-4 12.47%→3.97% (Aleithan 2024/2025). Removing leaks + strengthening tests: −27 pp Lite, −36.27 pp Verified (Xue 2026).

Reproducibility/variance. Top commercial entries closed/unverified (Aleithan 2024). SWE-bench-Live: 93 repos, Docker; big gap vs static (Zhang 2025). SWE-Bench Pro inherited tests: 32.4% judge disagreement vs 1.4% for DeepSWE hand-written verifiers (Huang 2026). Agent design introduces up to 6× performance variance for the same model across frameworks (Tian 2026, SWE-bench Mobile). ~61% of benchmarks focus narrowly on implementation (Wang 2025 SDLC survey).

Mitigations. SWE-rebench (21k+ fresh tasks; Badertdinov 2025); SWE-Bench Pro public/held-out/commercial splits (Deng 2025); DeepSWE original tasks in 91 repos never upstreamed (Huang 2026); LiveCodeBench timestamps (Jain 2024); Cross-Context Verification (Song 2026); ScienceAgentBench (Chen 2024); dynamic benchmarking with equivalent variants (Chen 2025); SWE-MERA, SWE-Compass, SWE-PolyBench.

Evidence on multi-agent vs single-agent is mixed and depends on task complexity, model capability and verification design (Gao 2025; Kim 2026; Saleh 2026).

Gains on complex tasks. AgentForge: 40.0% SWE-bench Lite, +26–28 pts over single-agent baselines via execution-grounded verification + role decomposition (Kumar 2026). Agyn: 72.2% on "SWE-bench 500" replicating team structure (Benkovich 2026). TheBotCompany self-organizing teams beat same-model single agents on the hardest ProjDevBench tasks; single agent more efficient on simple ones (Lyu 2026). README generation: single-agent pipeline matches multi-agent quality with 86% fewer tokens, 2× speed (Saleh 2026, "illusion of agentic complexity").

Diminishing benefit. 260-configuration controlled study finds a capability-saturation threshold beyond which more agents don't help; predicts multi-agent effect in 94% of SWE-bench Verified validation configs; single-agent baseline is the best predictor of whether coordination helps (Kim 2026, "Capable language models can outgrow the benefits of…"). Frontier long-context/tool-use gains remove original motivations (Gao 2025). Hybrid MAS/SAS request cascading +1.1–12% accuracy, −20% cost (Gao 2025). Waterfall-style multi-agent workflows: cleaner code but −37.8% (GPT-4o-mini) / −39.8% (DeepSeek-Chat) functional correctness; Claude-3.5-Haiku exception (Shafin 2025). Coordination breakdowns can severely degrade MAS (Gao 2025).

Orchestration/verification. He & Treude 2024 taxonomy of interaction models and planning styles. Three deficiencies: under-specification, coordination misalignment, inappropriate verification; SEMAP protocol reduces failures up to 69.6% (function-level dev) / 47.4% (Python vuln detection) (Mao 2025). Process-level verification not consistently helpful, high variance; LLM-as-judge > reward-based; trained judges > general LLMs (Venkataramani 2026, MASProve). VeriMAP planner-defined verification functions (Xu 2025). SagaLLM transactional validation (Chang 2025). Verification adds value when acceptance criteria are continuous, overhead when deterministic oracles exist (Lyu 2026).

Interaction histories grow unbounded; performance degrades well before the window limit (Cim 2026; Sun 2025; Semenov 2026). Dominant response is compaction (summarize + resume) with information loss, latency and safety risks (Cim 2026; Chen 2026 ×2).

Append-only problem / context rot. ReAct append-only histories → redundancy, semantic drift, reasoning collapse (Verma 2026; Liu 2025; Li 2026; Gao 2026). Accuracy drops of 14–85% with context length even with all info present ("context rot"; Chen 2026). Production systems (Claude Code, Codex, LangChain, LlamaIndex) use LLM summarization at window limits; Anthropic recommends compaction at 5–20k tokens for some workloads (Cim 2026; Chen 2026).

Compaction limitations. 90–99% token reduction but always lossy (Cim 2026); salience mismatch (Semenov 2026); causal structure destroyed; synchronous compaction stalls tens of seconds; compression-induced hallucination; governance decay: over 1,323 episodes, violation rates rose from 0% (policy in context) to 30% after compaction, up to 59% for some models; 0% when constraint survives summary, 38% when dropped (Chen 2026).

Agent-initiated / learnable. Context-as-a-tool: CAT framework; SWE-Compressor 57.6% on SWE-Bench Verified (Liu 2025). SWE-MeM: when/what/how to compress; 60.2% Verified with a 30B model (Gao 2026). CompactionRL: +7.0 pts for GLM-4.5-Air → 66.8% Verified (Li 2026). Context-Folding: 10× smaller active context matching ReAct (Sun 2025). AgentFold look-back folding (Ye 2025). ACON (Kang 2025).

Lossless/external memory. Memex indexed external archive, RL-trained (Wang 2026); ACM manage_context/query_memory tools (Li 2026); CWL dependency-graph eviction (Semenov 2026); Git-Context-Controller COMMIT/BRANCH/MERGE — >13% relative improvement, >80% on SWE-Bench Verified (Wu 2025); SimpleMem +26.4% F1, up to 30× fewer tokens (Liu 2026).

Validation / external management / open problems. Slipstream async compaction + judge: +8.8 pp accuracy, −39.7% latency on SWE-Bench Verified & BrowseComp (Chen 2026). AdaCoM external manager for frozen agent — fidelity–reliability trade-off (Yi 2026). Rate–distortion view unifying KV-cache eviction to long-term memory; repeated compaction almost never measured (Colaço 2026). Constraint Pinning restores 0% violations. Coding agents as long-context processors via file system: +17.3% avg over SOTA (Cao 2026). MEM1; Memory-as-Action.

Three facets: sandboxing/isolation; permission/access-control models; human oversight (Yan 2025; Navneet 2025; Marchand 2026).

Sandboxing. Docker/OCI most common (Marchand 2026; Piao 2025; Kumar 2026). Fault-tolerant transactional sandbox: 100% interception of high-risk commands, 100% rollback, 14.5% overhead/transaction (Yan 2025). Gemini CLI sandbox requires interactive auth that breaks headless loops (Yan 2025). Frontier LLMs can find/exploit sandbox escapes (misconfig, privilege, kernel) (Marchand 2026). Survey of 39 execution-security papers: isolation architectures and capability models almost never compared on a shared benchmark; policy-enforcement failure rates 69–98% vs real denylists (Rashidi 2026, "Balkanization"). Hardware-gated system for LLM-written ARTIQ (Wang 2026). seccomp/cgroups/namespaces (Piao 2025).

Permissions. Classical principles (defense-in-depth, least privilege, complete mediation, psychological acceptability) transfer and are cheaper than ad hoc scaffolding (Zhang 2025 "LLM agents should employ security principles"; Winninger 2026). Constrained substrate improved backdoor-detection recall 54.5%→90.9% with a small reviewer model (Winninger 2026). AgentSpec DSL prevents >90% unsafe executions at ms overhead (Wang & Poskitt 2025). Capability-enhanced MCP with structured labels (Doshi 2026). LlamaFirewall (Chennabasappa 2025).

Human oversight. Autonomy levels L1–L2 vs L4–L5 shift safeguards from procedural to architectural (Su 2025 survey). Token-gated authorization bound to exact tool-call contents (Wang 2026). AgentBay hybrid human takeover, >48% task-success improvement (Piao 2025). Scalable-oversight bottleneck: human review is the limiting factor (Winninger 2026). SAFE-AI framework taxonomy suggestive/generative/autonomous/destructive actions (Navneet 2025). 547 confirmed safety failures: >65% during bug-fixing and setup/config; dominant risks constraint violations, destructive ops, auth bypass, deception (Hasan 2026, "What breaks when LLMs code").

PR outcomes. 567 Claude Code PRs / 157 projects: 83.8% merge rate vs 91.0% human; 54.9% merged unmodified; revision extent after review does not differ (Watanabe 2025). 7,156 PRs from five agents: task type dominates — documentation 82.1% vs new features 66.1%; Codex 59.6–88.6% across 9 categories; Claude Code 92.3% docs / 72.6% features; Cursor 80.4% fixes; Devin +0.77%/week trend, underperforms on fixes; no agent dominates (Pinna 2026). Only 35.7% of rejected PRs reflect clear agentic failures; 31.2% workflow constraints; 33.1% no observable rationale (Peralta 2026).

Productivity. Field experiment 1,974 devs at Microsoft/Accenture: Copilot +12.92–21.83% PRs/week (MS), +7.51–8.69% (Accenture) (Cui 2024). 95 programmers: Copilot 55.8% faster (Martin-Lopez 2026 recounts). Cursor controlled experiment with experienced OSS devs: limited benefit (Agarwal 2026). Velocity gains front-loaded, heterogeneous; prior AI-IDE repos show minimal/short-lived gains; Cursor gains concentrate in first two months (Agarwal 2026; Pinna 2026). Copilot vs agent user study (Chen 2025). Ownership trade-off: >2× completeness but −12.5% ability to answer questions about own code (Martin-Lopez 2026). Core vs peripheral devs differ in delegation and CI use (Cynthia 2026).

Quality/debt. Static-analysis warnings +18%, cognitive complexity +39% after agent adoption (Agarwal 2026). Agent code churns more (Popescu 2026), retained less (Mazloomzadeh 2026); smaller, more localized edits, with Claude Code/Codex more variable than Devin/Cursor/Copilot (Ogenrwot 2026). Agentic refactoring in 26.1% of commits, low-level consistency edits (Horikawa 2025). Security PRs ~4% of activity, lower merge rate, longer latency (Siddiq 2026).

Adoption/workflow. 22.2–28.7% of GitHub projects use coding agents by early 2026 (Robbes 2026). AIDev: 932,791 agent PRs, 116,211 repos, 72,189 devs (Li & Hassan 2026). Collaborator (Cursor, Devin, Copilot) vs Assistant (OpenAI, Claude) partitioning; merge governance stays human (Chung 2026). Failed PRs larger, more files, CI failures (Ehsani 2026); test failures and duplicate resolution main causes (Alam 2026). Code-review agents as gatekeepers: 45.2% merge, 23 pp below human-only (Chowdhury 2026). PR communication (Watanabe & Tsuchida).

Surveys decompose coding-agent architecture into layered taxonomies: control loops, planning, tool interfaces, verification (Rombaut 2026; Ning 2026; Guo 2026). Field moved from capability-based to source-code-level, harness-centric frameworks.

Control loop. Foundational dimension (Rombaut 2026): five composable primitives — ReAct, generate-test-repair, plan-execute, multi-attempt retry, tree search; 11 of 13 agents compose several; spectrum from fixed pipeline to MCTS. Agent Loop weaknesses: implicit step dependencies, unbounded recovery loops, mutable execution history → Structured Graph Harness with explicit static DAG, three layers, immutable plan versioning (Wei 2026). Agentic Programming / LLM-as-code: deterministic program owns control flow, LLM invoked only for reasoning; context bounded by call depth (Qi 2026).

Planning. Four types by locus of harness control: linear decomposition, structure-grounded, search-based, orchestration-based (Ning 2026 "Code as Agent Harness"). Huang 2024 planning survey taxonomy. AdaCoder two-phase adaptive planning (Zhu 2025). CoT still most common in code generation (Liu 2024).

Tool interface/environment. Tool definition via JSON-schema function calling, inline prompt descriptions, or custom text formats; tool counts 0–37 across scaffolds; static vs dynamic tool discovery (Rombaut 2026). Typed tool schemas as primary control surface (Xu 2026). Dimensions converge where external constraints dominate (tool categories, edit formats, execution isolation) and diverge on context compaction (seven strategies across 13 agents), state management, multi-model routing (Rombaut 2026).

Verification. From autonomous test execution to multi-agent review with formal checks (Liu 2024; Agents4PLC). Five-level verification ladder; 70% of analyzed loops in autonomous zone (Macedo 2026, "Stop handholding your coding agent"). SagaLLM transactional (Chang 2025). Structured Cognitive Loop with Regulation layer: zero policy violations (Kim 2025).

Harness as performance variable. Up to 40× tokens per solved task, 0–8 pp within-model pass-rate spread; failure fingerprints replicate across models within a harness (Vats & Golev 2026). Six coupled runtime responsibilities — observation, context, control, action, state, verification (Guo 2026). Paradigm progression: prompt optimization → context engineering → harness engineering → agent-native training / model–harness co-evolution (Guo 2026). Six-component harness tuple; seven-layer ETCLOVG taxonomy (Guo 2026). "Code as agent harness": interfaces / mechanisms / scaling (Ning 2026).