Date: 2026-10-06 · Plan §7 Task 1 (evidence phase)
Derived from the store docs/research/ai-in-empirical-se/sources.json (219 records: 53 CONFIRMED, 26 MINOR-FIX, 139 UNCHECKED, 1 WRONG) via docs/research/ai-in-empirical-se/SYNTHESIS.md. Record ids are in backticks; the verification status shown is the store's own.
Every row is a candidate until it is verified with /research:add-paper. Store verification is not the citation gate: no row may be cited in the research write-up until it is in the verified citation matrix.
| Source | What it already establishes | Limitation/gap | Relation to the research | Use in research? |
|---|---|---|---|---|
| Gomes et al. 2026 [gomes-2026-llm-assisted-ese-slr] (MINOR-FIX) | SLR of 50 ESE studies, 69 LLM-assisted tasks. Use clusters in MSR and data processing; integration is automation-oriented; reporting is often incomplete | Maps use, not how the process is governed; no account of delegated decisions or gates | Anchor: shows AI is already a participant in ESE and that reporting lags | Yes — cite |
| Baltes et al. 2025 [baltes-2025-llm-guidelines-ese] (UNCHECKED) | 8 guidelines across 7 study types, including session traces and validating LLM outputs against human judgement | Per-study disclosure checklist; no authority bounds, typed decision records or gate semantics | The reporting floor the research builds on; refutes "model + prompt only" | Yes — position against |
| llm-guidelines.org 2026 [llmguidelines-2026-session-traces] (UNCHECKED) | Agentic studies should report HITL approval and rejection decisions, plus tool-call order | "Should"-level raw logs; does not say what a human verifies, by what criteria, or where gates sit | Closest pre-emption of the gap sentence; the research sharpens it into designed gates | Yes — position against |
| Si et al. 2025 [si-2025-ideation-execution-gap] (CONFIRMED) | Stage 1: the advantage of LLM research ideas shrinks or reverses once 43 experts execute them | NLP, not SE; adoption decisions are not recorded | Supports a gate on accepting AI-proposed questions | Maybe |
| Theodoro et al. 2026 [theodoro-2026-lri-co-assistant] (CONFIRMED) | Stage 1 SE tool: accept/edit/discard step and a Go/Pivot/Abort decision | Whether decisions are logged as provenance is not stated | Existing SE instance of a semantic gate at question design | Maybe |
| Kitchenham et al. 2026 [kitchenham-2026-guest-genai-slr-guidelines] (UNCHECKED) | Stage 2: GUEST covers planning, conducting and reporting GenAI-supported SLRs; human oversight required | One stage only; reporting-oriented | Closest SE process-level comparator; the research generalises across the lifecycle | Yes — position against |
| Flemyng et al. 2025 [flemyng-2025-ai-evidence-synthesis-position-statement] (UNCHECKED) | Report any AI use that makes or suggests judgements, plus stages affected and justification; authors accountable for the decision to use AI | Evidence synthesis only; disclosure, not engineered process | Mature precedent outside SE; shows disclosure already extends past model + prompt | Yes — cite |
| Wang et al. 2025 [wang-2025-reassessing-llm-boolean-query] (UNCHECKED) | Stage 2: a reproduction diverged because intermediate steps (validation, formatting, example choice) were unreported | Diagnoses the failure; proposes no recording scheme | Direct evidence that model + prompt is insufficient | Yes — cite |
| Ribeiro & Hida 2026 [ribeiro-2026-slr-genai-graduate-course] (CONFIRMED) | Stage 2: names "excessive delegation" and "superficial validation"; recommends decision records | Educational case study; decision records not specified | Names the research's concern, so the research supplies the specification | Yes — cite |
| De Martino et al. 2025 [de-martino-2025-primes2-llm-msr] (UNCHECKED) | Stage 3: PRIMES 2.0 has 6 stages and 23 substeps mapped to 9 threats and 25 mitigations | MSR only; tracks outputs and prompts, not authority or gates | Stage-specific prior art the research's patterns must interoperate with | Yes — position against |
| Ahmed et al. 2025 [ahmed-2025-llm-replace-manual-annotation-se] (UNCHECKED) | Stage 3: LLM–human agreement is equal or close to human–human on some tasks; gives delegation criteria | The decision to delegate is not itself recorded or reported | Complicates the thesis; its criteria are a delegated decision to record | Yes — position against |
| Härtel 2026 [hartel-2026-agentic-repository-mining] (UNCHECKED) | Stage 3: agents label 4,943 MSR items while retrieving their own context | The context an agent chooses is unrecorded and in no reporting standard | Concrete case of agentic participation in ESE | Maybe |
| Jeldtoft & Yousef 2026 [jeldtoft-2026-auditable-llm-ta-workflow] (UNCHECKED) | Stage 4: traceability from material to outputs; LLM inference constrained to interpretive tasks | Thematic analysis only | Closest single-stage prior art for bounded authority plus traceability | Yes — position against |
| Prahl 2025 [prahl-2025-arc] (CONFIRMED) | Stage 4: a pre-analysis checklist that makes the delegation decision a reviewable record | Qualitative coding in health research; one decision point | Direct precedent for delegated-decision records | Yes — cite |
| Jowsey et al. 2025 [jowsey-2025-reject-genai-reflexive] (UNCHECKED) | Stage 4: 419 researchers reject GenAI for reflexive thematic analysis | Applies to Big-Q reflexive work | Forces the research to scope its claims, or to treat "do not delegate" as a valid gate outcome | Maybe |
| Fogliato et al. 2026 [fogliato-2026-many-ai-analysts] (UNCHECKED) | Stage 5: AI analysts reproduce many-analyst dispersion; persona or LLM steers results | Decisions reconstructed afterwards; proposes multiverse reporting | Delegated analytic decisions drive conclusions | Yes — cite |
| Cui & Alexander 2026 [cui-2026-same-prompt-different-outcomes] (MINOR-FIX) | Stage 5: the same configuration gives considerably varying results across 480 attempts | Recommends repeated runs only | Supports executable reproducibility and run records | Yes — cite |
| Ifargan et al. 2024 [ifargan-2024-data-to-paper] (UNCHECKED) | Stage 6: information tracing chains results to methods and data; human co-pilot | Co-piloting framed as a capability; no authority bounds or gate reporting | Main provenance prior-art risk | Yes — position against |
| Luo et al. 2025 [luo-2025-hidden-pitfalls-ai-scientist] (MINOR-FIX) | Stage 6: trace logs and code catch failures the paper hides; recommends mandating logs | Calls for logs, not for structured governance | The research's increment is structure (decisions, bounds, gates) over log dumps | Yes — position against |
| Ding et al. 2026 [ding-2026-autonomous-research-agents-verification-gap] (CONFIRMED) | Stages 6/8: 38% of 24 systems release seeds or traces; no externally validated in-loop oracle | ML AI-scientists, not ESE | Quantifies the verification gap | Yes — cite |
| Angermeir et al. 2025 [angermeir-2025-reproducibility-commercial-llm-ese] (UNCHECKED) | Stage 7: of 18 ICSE studies, 5 executable and 0 fully reproducible | Artifact-level; no account of process | Hard SE numbers for executable reproducibility | Yes — cite |
| EC 2026 [ec-2026-living-guidelines-genai-research] (UNCHECKED) | Stage 7: lit review and hypothesis generation count as substantial use; asks how AI "affected the research process"; FAIR prompts and outputs | Narrative disclosure; no gates or authority | The most process-aware policy; supports the framing | Yes — cite |
| Ralph et al. 2020 [ralph-2020-acm-sigsoft-empirical-standards] (UNCHECKED) | Stage 7: 19 method standards | No LLM or AI standard (checked 2026-10-06) | Gap evidence and a natural target for the six requirements | Yes — cite |
Springer Nature springernature-ai-guidance (UNCHECKED) |
Stage 7: book authors disclose LLM use in the acknowledgements or preface | Narrative only; the copy-editing rule is unresolved | Governs this research's own disclosure | Maybe |
| Souza et al. 2025 [souza-2025-prov-agent] (CONFIRMED) | Stage 8: W3C-PROV model relating agent prompts, responses and decisions to workflow outcomes | e-Science workflows; records decisions but not their authority | Reusable substrate for durable provenance | Yes — cite |
| Zhu et al. 2026 [zhu-2026-hler-human-oversight] (MINOR-FIX) | Stage 8: three human gates plus deterministic data work cut critical failures from 72% to 16% | Economics, not ESE; preprint | Strongest empirical support, and closest prior art, for bounded authority and gates | Yes — position against |
| Krebs 2026 [krebs-2026-fai2r] (UNCHECKED) | Stage 8: "no parentless claim"; verification rungs only humans may grant; CI gating | Single-author preprint; not tied to empirical methods | Close prior art for control-plane-style governance | Yes — position against |
| Grunde-McLaughlin et al. 2026 [grundemclaughlin-2026-overseeing-agents] (UNCHECKED) | Stage 8: better traces raised confidence, not accuracy | Generic agent oversight | Complicates the thesis: provenance alone is not rigor, so gates need criteria | Yes — cite |
| Cao et al. 2025 [cao-2025-otto-sr] (MINOR-FIX) | Complicating: agentic SR beats dual human review on screening and extraction; reproduced 12 Cochrane reviews | Changed significance in 3 reviews; relied on dual human review of discrepancies; the "two days" figure is unverified | Strongest capability counter-example; argue accountability, not incompetence | Yes — position against |
| Trinkenreich et al. 2026 [trinkenreich-2026-pulse-genai-se-research] (UNCHECKED) | Complicating: of 457 SE researchers, use is mostly writing and early-stage; researchers call for governance | Self-report survey | Concede the agentic scenario is ahead of practice; supports the demand for governance | Yes — cite |
Gap statement (candidate)
- SE guidance already asks for model versions, prompts, session traces and human validation of LLM outputs ([baltes-2025-llm-guidelines-ese]). For agentic studies it recommends reporting human approval and rejection decisions ([llmguidelines-2026-session-traces]). The gap is therefore not disclosure itself.
- Process-level governance exists only stage by stage: SLRs ([kitchenham-2026-guest-genai-slr-guidelines]), MSR ([de-martino-2025-primes2-llm-msr]) and thematic analysis ([jeldtoft-2026-auditable-llm-ta-workflow]). Whole-workflow gated provenance exists only outside ESE, in 2026 preprints ([zhu-2026-hler-human-oversight], [krebs-2026-fai2r]).
- No ESE source specifies delegated decisions (which decision, to what, accepted by whom), authority bounds or human semantic gates (what is checked, against what criteria, where in the lifecycle) as reportable, reviewable provenance. The closest are [prahl-2025-arc], which covers one decision in qualitative coding, and [lin-2026-llm-se-security-assurance-survey], which names "agent authority" for evaluating agents rather than for research.
- Logging alone is not the answer: richer traces raise overseers' confidence without raising their accuracy ([grundemclaughlin-2026-overseeing-agents]). That is why gates must be semantic and designed in advance.
- Candidate sentence: Current SE guidance asks for prompts, traces and human validation, but treats them as per-study disclosures; it does not specify delegated decisions, authority bounds or human semantic gates as reportable, reviewable provenance across the empirical lifecycle.