AI and Agents across the Empirical-SE Lifecycle: Synthesis

This is a derived digest of the store, not a source of truth. Every claim below cites a record id in sources.json. Where a number comes from an UNCHECKED record it is flagged as such where it matters. Claims that a record itself marks as coming from search snippets only are never used as load-bearing. No record here is citable in the research write-up until it has gone through /research:add-paper.

Id note. The memos sometimes use pre-merge ids, such as "bertran-2026-many-ai-analysts", "demartino-…" or "angermeir-2026-…" (quoted here, not cited). This digest uses only the merged ids in sources.json, for example [fogliato-2026-many-ai-analysts], [de-martino-2025-primes2-llm-msr] and [angermeir-2025-reproducibility-commercial-llm-ese].

1. The lifecycle map

The anchor SLR, [gomes-2026-llm-assisted-ese-slr] (MINOR-FIX), maps 50 primary studies in 12 SE venues covering 2020–2025. It finds 69 LLM-assisted tasks, which cluster in repository mining and controlled experiments and focus on classification, filtering and evaluation. LLM use sits mostly in the data-processing and analysis phases. Integration is "largely automation-oriented, with limited decision-support use", and reporting is often incomplete. The rest of the store fills in the stages that SLR touches lightly.

Stage 1: question formulation and study design. - What AI does in SE: it proposes problem attributes and refines research questions ([pereira-2025-ai-agents-research-problem-formulation], a vision paper). One tool runs a five-phase research-idea workflow in which every AI suggestion is accepted, edited or discarded, ending in a Go/Pivot/Abort decision ([theodoro-2026-lri-co-assistant], CONFIRMED). AI can also stand in for participants at the design stage ([gerosa-2024-ai-substitute-human-subjects]), and GPT-4 can plan analyses for published ESE studies ([liang-2024-gpt4-replicate-ese]). - Evidence outside SE: this is stronger. LLM ideas were rated more novel than expert ideas ([si-2024-llm-novel-research-ideas]). Once 43 experts executed the ideas, that advantage shrank or reversed ([si-2025-ideation-execution-gap], CONFIRMED). Agentic systems design experiments themselves ([baek-2025-researchagent], [schmidgall-2025-agent-laboratory]), and one runs human-participant studies end to end ([wehr-2026-closing-empirical-loop], CONFIRMED). - Maturity: vision and tool level in SE. No SE study evaluates AI-generated research questions (agent-a memo; consensus memo q1 "effectively empty").

Stage 2: literature reviews (SLRs, mapping studies, MLRs). - Screening has the most evidence in SE. Single-LLM recall reached 98% at 27% precision ([thode-2025-llm-selection-phase-slr], CONFIRMED). Some LLM and prompt combinations match human screeners but are not significantly more accurate ([huotala-2024-llm-screening-sr-ease], CONFIRMED). A 34,528-study benchmark concludes LLMs are "not yet recommended" for automating title-abstract screening ([huotala-2025-sesr-eval]). Disagreement analysis finds κ of 0.52–0.77 ([mantyla-2026-llm-screening-disagreements]). - Variability is a recurring finding. Screening is non-deterministic even at temperature zero ([hida-2026-llm-variability-screening]), and batching changes decisions ([hida-2026-class-imbalance-batch-screening], CONFIRMED). - Other review steps have been studied too: search strings ([marassi-2025-llm-search-string-generation]), mapping studies ([petersen-2024-interactive-llm-mapping-studies]), fully automated SLR pipelines ([sami-2024-multi-agent-slr-system]), and agentic rapid reviews ([nyyssola-2026-llm-rapid-reviews-tool-discovery]). - Outside SE: a 63-study review shows good agreement on mechanical tasks and weak agreement on judgement tasks ([laignelot-2026-llm-systematic-review-sr], CONFIRMED). Agentic SR pipelines exist ([cao-2025-otto-sr], MINOR-FIX). - Maturity: empirical evaluation for screening, emerging guidance (GUEST, ARISMA, RAISE; §3), and no methodology papers for AI-supported multivocal reviews (agent-a memo).

Stage 3: repository mining and data curation. - LLMs replace manual annotation. In SE, LLM–human agreement is equal or close to human–human agreement on some tasks ([ahmed-2025-llm-replace-manual-annotation-se]). Outside SE, ChatGPT beat crowd workers by about 25 percentage points ([gilardi-2023-chatgpt-outperforms-crowd-workers], CONFIRMED). - LLMs also label MSR artifacts directly: issues ([colavito-2025-benchmarking-llm-labeling]) and code-review comments, where zero-shot macro-F1 was 0.36–0.37 ([caglar-2026-code-review-comment-classification]). - Agents now curate data autonomously. Bash-exploring agents produced 4,943 classifications while retrieving their own context ([hartel-2026-agentic-repository-mining]). One EMSE pipeline uses LLM labels with no human input at all ([senthilkumar-2026-synthcore-llm-labels]). - Process frameworks exist: PRIMES ([de-martino-2025-primes-llm-msr-framework]), PRIMES 2.0 with 6 stages, 23 substeps, 9 threats and 25 mitigations ([de-martino-2025-primes2-llm-msr]), and OLAF ([imran-2026-olaf-llm-annotation]). - Maturity: empirical evaluation plus SE-specific process guidance. Agentic MSR is just emerging, in 2026.

Stage 4: qualitative analysis. - Deductive coding is well studied: - Cohen's κ 0.57 for inductive coding and Fleiss' κ 0.46 for deductive coding ([zhang-2025-qualigpt-irr], CONFIRMED) - 160 repeated runs per text to measure variation ([tai-2024-llm-aid-textual-analysis], CONFIRMED) - a decision rule for when to delegate coding ([chew-2023-llm-assisted-content-analysis]) - Inductive thematic analysis in SE: experts preferred LLM codes 61% of the time, but the LLMs missed latent meaning ([martinez-montes-2025-llm-thematic-analysis-se]). - The SE community is active: a mapping study ([leca-2025-llm-qualitative-analysis-ese-mapping]), a 25-person ISERN workshop ([alves-2026-llm-thematic-analysis-isern-workshop]), a TOSEM position ([ernst-2026-genai-no-silver-bullet-qualitative]) and an antipattern catalogue ([hoda-2026-antipatterns-ai-qda], CONFIRMED). - Auditability work is very recent: [jeldtoft-2026-auditable-llm-ta-workflow], [nyaaba-2026-hacita-interpretive-authority] and [prahl-2025-arc] (CONFIRMED). - Maturity: empirical evaluation for deductive coding; guidance is forming (2025–26 workshops).

Stage 5: quantitative analysis, statistics and simulation. - Agents run analyses end to end, and their conclusions depend on delegated choices. AI analysts reproduce the dispersion of human many-analyst studies, and the choice of persona or LLM steers the results ([fogliato-2026-many-ai-analysts], PNAS, UNCHECKED). Identical configurations give considerably different results across 480 attempts ([cui-2026-same-prompt-different-outcomes], MINOR-FIX). Configuration choices yield wrong conclusions for about 31% of hypotheses ([baumann-2025-llm-hacking]). - Capability on realistic tasks is 25–65%: - 25% ([majumder-2024-discoverybench], CONFIRMED) - 64.83% ([zhu-2024-statqa]) - 34.12% ([jing-2024-dsbench]) - Synthetic participants are contested. Supporters: [argyle-2023-out-of-one-many], [aher-2023-turing-experiments]. Critics: [bisbee-2024-synthetic-survey-perils], [cummins-2025-analytic-flexibility-silicon]. One study reconciles both sides: aggregate fidelity is not individual-level fidelity, with only 3.05% of respondent-specific variance explained ([ahn-2026-item-mean-surrogates], CONFIRMED). - SE has a specific version of the problem: AI-written responses now enter SE surveys through the participants themselves ([desouzasantos-2026-fraudulent-ai-se-surveys]). - Maturity: benchmarks plus empirical evaluation; low to medium.

Stage 6: artifact generation and agentic science. - AI-scientist systems produce whole papers. Systems include [lu-2024-ai-scientist] (CONFIRMED), [yamada-2025-ai-scientist-v2], [yamada-2026-ai-scientist-nature], [mitchener-2025-kosmos] (CONFIRMED) and [ifargan-2024-data-to-paper]. A generated manuscript passed the first review round of a workshop with a 70% acceptance rate ([yamada-2026-ai-scientist-nature]). - Reproduction and replication agents are improving: - [starace-2025-paperbench]: 21.0% - [siegel-2024-core-bench]: 21% on the hardest tasks - [alizadeh-2026-socsci-repro-bench]: "a large share" of social-science findings reproduced - SE-specific artifact agents check replication-package quality and evaluate ACM badges: [mbida-2026-agentic-replication-package-quality], [wu-2026-artifactcopilot] (CONFIRMED) and [baek-2026-artisan-table-reproduction]. - Critiques find verification is the bottleneck: - [zhu-2025-ai-scientists-implementation] (CONFIRMED) - [ding-2026-autonomous-research-agents-verification-gap] (CONFIRMED) - [beel-2025-evaluating-sakana-ai-scientist]: 42% of experiments failed - [chen-2025-mlr-bench]: about 80% of cases had fabricated or invalidated results - Maturity: working tools and benchmarks. Computational reproduction is maturing fast; open-ended AI research is fluent but unreliable.

Stage 7: reporting and reproducibility. - SE has real guidance: [baltes-2025-llm-guidelines-ese] with its living site [llmguidelines-2026-session-traces], the precursor [wagner-2025-evaluation-guidelines-llm-studies], and a prompting guideline ([korn-2026-reporting-llm-prompting-ase], CONFIRMED). - Audits show reproduction failing. Of 18 ICSE studies attempted, 5 were executable and none fully reproduced ([angermeir-2025-reproducibility-commercial-llm-ese]). An audit of 640 papers found that badges do not guarantee reproducibility ([siddiq-2025-llm4se-reproducibility-crisis]). Non-determinism persists at temperature 0 ([ouyang-2025-nondeterminism-chatgpt]). - The ACM SIGSOFT Empirical Standards have no AI standard ([ralph-2020-acm-sigsoft-empirical-standards], checked 2026-10-06). - Maturity: guidance exists; it is versioned but not enforced by any venue found (agent-d memo).

Stage 8: human-in-the-loop, provenance and audit. - Provenance standards are mature: [w3c-2013-prov-overview] (2013) and [souza-2019-prov-ml] (CONFIRMED). Agent-specific provenance is new (2025–26): [souza-2025-prov-agent] (CONFIRMED), [wu-2026-agenttrails] and [rosen-2026-execution-lineage]. - Gated agentic-research architectures appear in 2026 preprints, mostly outside SE: [zhu-2026-hler-human-oversight], [krebs-2026-fai2r], [dong-2026-crossaudit] and [jha-2026-paper-pilot]. - Oversight research is sceptical that nominal human-in-the-loop works ([grundemclaughlin-2026-overseeing-agents]; [mitchell-2026-agents-push-humans-out], CONFIRMED). - Maturity: standards are mature; agentic provenance and gating are at tool or early-empirical level, and none of it comes from SE research methods.

Compact table

2. Rigor, validity and reproducibility, stage by stage

Stage 1. - Novelty judged before execution does not survive execution. LLM ideas lost significantly more score than human ideas once executed, and on many metrics the rankings flipped ([si-2025-ideation-execution-gap], CONFIRMED). The SE record says reproducibility becomes contingent on model updates and prompt context, and that core constructs ("developer", "artifact") are challenged ([treude-2025-genai-empirical-se-paradigm-shift], MINOR-FIX). - Expert review catches subtle errors. GPT-4 surfaced correct assumptions but made many small implementation errors that reflect a lack of SE knowledge ([liang-2024-gpt4-replicate-ese]). - No source treats the acceptance of an AI-proposed question as a recorded decision (agent-a memo).

Stage 2. - Accuracy metrics mislead. Apparent human-par accuracy was inflated by chance agreement and class imbalance ([khraisha-2024-gpt4-systematic-review-screening], MINOR-FIX). Only 10% of 29 SE screening papers report MCC, and a model optimised for accuracy lost 63.3% of the relevant evidence ([madeyski-2026-llm4screenlit], UNCHECKED). - Operational parameters change outcomes: batching ([hida-2026-class-imbalance-batch-screening], CONFIRMED), prompt wording and rating scale ([dennstadt-2024-llm-title-abstract-screening], CONFIRMED), and run-to-run inconsistency in extraction ([oami-2026-llm-data-extraction-prompting], CONFIRMED). - Unrecorded steps blocked reproduction. A reproduction of LLM Boolean-query generation diverged from the original because query validation, output formatting and the choice of chain-of-thought examples went unreported ([wang-2025-reassessing-llm-boolean-query], UNCHECKED). This is the most direct evidence in the store for the research's thesis. - Case-study observation: "excessive delegation" and "superficial validation" ([ribeiro-2026-slr-genai-graduate-course], CONFIRMED).

Stage 3. - Validation is task by task. LLM annotation is "highly contingent on dataset and task type" ([pangakis-2023-automated-annotation-validation]). Non-determinism can fall below reliability thresholds ([reiss-2023-chatgpt-annotation-reliability]). There is a formal statistical test for justifying the substitution of human annotators ([calderon-2025-alternative-annotator-test], CONFIRMED). - Threats: closed models, leakage and non-determinism ([sallou-2024-breaking-the-silence]). Contamination averages 0.7–4.8% by language but reaches 100% for QuixBugs ([zhou-2025-lessleak-bench]). - Analysis choices change MSR conclusions. Trend direction flips under equally defensible analysis choices ([agarwal-2026-3100-opinions-code-review], CONFIRMED). The "ground truth" behind agentic labelling is itself context-limited ([hartel-2026-agentic-repository-mining]).

Stage 4. - Agreement is not validity. Agreement with human annotators leaves construct validity untouched, so validation should target the process ([pita-2026-correct-codes-wrong-reasons]). High run-to-run κ (above 0.8) is self-consistency, not validity ([jain-2025-multi-llm-thematic-analysis-kappa], CONFIRMED). Prompt strategy changes agreement and causes systematic label bias of up to 5.25× ([alshaikh-2026-prompt-llm-qual-coding-psych-safety]). - Every SE source keeps interpretive authority with humans: [leca-2025-llm-qualitative-analysis-ese-mapping], [alves-2026-llm-thematic-analysis-isern-workshop], [dunivin-2025-scaling-hermeneutics] (CONFIRMED), [schroeder-2025-llms-qualitative-uses-tensions] (CONFIRMED; "urgent lack of norms and tooling").

Stage 5. - Conclusions depend on delegated analytic decisions. - Many-analyst dispersion is reproduced and can be steered ([fogliato-2026-many-ai-analysts]). - In 6 of 11 datasets, a single run's affirmative conclusion was not stable ([rewolinski-2026-sanity-checks-agentic-ds]). - A confirmatory prompt flips verdicts from 10% to 90% support while leaving the coefficients unchanged ([alizadeh-2026-coding-agents-social-science]). - LLMs make characteristic errors. They mostly choose tests whose assumptions do not hold ([zhu-2024-statqa]). - Process-level remedies: - preregister, then run on the next model released ([thomas-2026-preregister-next-llm], MINOR-FIX) - a preregistration template for AI-agent experiments ([vaccaro-2026-preregistration-ai-agents]) - protocol enforcement plus deterministic computation ([lee-2026-autometa], MINOR-FIX) - reproducibility-enhancing workflow prompts ([zeng-2025-airepr], CONFIRMED) - Synthetic data drifts. The same prompt gave significantly different results over three months ([bisbee-2024-synthetic-survey-perils]).

Stage 6. - The final paper hides failures. - Trace logs and code detect failures far better than the paper does ([luo-2025-hidden-pitfalls-ai-scientist], MINOR-FIX). - Agents reach right-looking results through wrong mechanisms ([eulig-2026-correct-answer-wrong-mechanism]). - With data missing, all seven models synthesise data rather than acknowledge infeasibility ([yang-2026-sciintegrity-bench], CONFIRMED). - Automated review accepted work that humans found flawed: 59% of accepted automated reviews contained fabricated or unsupported claims ([gaddipati-2026-mlreplicate], CONFIRMED). - 38% of 24 runnable systems release seeds or traces, and none demonstrates an externally validated in-loop oracle ([ding-2026-autonomous-research-agents-verification-gap], CONFIRMED).

Stage 7. - Reproducibility audits (above) give hard numbers: [angermeir-2025-reproducibility-commercial-llm-ese], [siddiq-2025-llm4se-reproducibility-crisis]. - Reporting practice diverges from what reviewers expect on version disclosure, prompt justification and threats to validity ([korn-2026-reporting-llm-prompting-ase], CONFIRMED). - Single-output reporting hides variance ([ortin-2026-safellm4se]). - One recommendation set separates data paths: keep the LLM off the data path where possible, preserve LLM-generated artifacts, and verify by methods suited to the LLM's role ([dunn-2026-reproducible-llm-analyses], CONFIRMED).

Stage 8. - Better traces improved speed and confidence, not accuracy ([grundemclaughlin-2026-overseeing-agents]). - Automation bias is the named failure mode of nominal oversight ([eu-2024-ai-act-article-14], an analogy only; [laux-2025-automation-bias-ai-act], whose claim is snippet-sourced). - Gates plus deterministic computation measurably reduce failures. Critical failures fell from 72% to 16% over 280 runs ([zhu-2026-hler-human-oversight], MINOR-FIX). Evidence-locked rules cut fabricated citations from up to 25% to zero ([jha-2026-paper-pilot], CONFIRMED).

3. Reporting and reproducibility guidance that already exists

The draft gap sentence ("disclosing model + prompt captures a sliver") understates existing guidance. Here is what each guideline already asks for:

SE-specific. - [baltes-2025-llm-guidelines-ese] (UNCHECKED; accepted in EMSE per the record). Eight guidelines across seven study types: 1. declare LLM usage and role 2. report model versions, configurations and customizations 3. document system and prompt design beyond the model 4. report session traces (interaction logs and runtime traces) 5. use suitable baselines, benchmarks and metrics 6. include an open LLM as a baseline 7. validate LLM outputs against human judgment 8. articulate limitations and mitigations

The study types include LLMs as annotators, as judges and for synthesis, which are uses inside the research process. - [llmguidelines-2026-session-traces] (UNCHECKED; version 2026.09). - Researchers should provide full interaction logs. - For agentic systems, they should report "human-in-the-loop feedback, approval or rejection decisions, and iterative refinements". - For tool-using agents, they report each tool call's name, arguments, result and ordering. - It notes that non-determinism persists at temperature 0.

This is the strongest pre-emption of the draft gap sentence. - [wagner-2025-evaluation-guidelines-llm-studies] (UNCHECKED). Explicitly scopes in studies that use LLMs "as part of the research process". - [korn-2026-reporting-llm-prompting-ase] (CONFIRMED). A prompting-report guideline built from about 300 papers and 105 PC members. - [kitchenham-2026-guest-genai-slr-guidelines] (GUEST; UNCHECKED). Process-level guidance for planning, conducting and reporting GenAI-supported SLRs. It holds that GenAI "requires human oversight and is not currently capable of unsupervised systematic studies". It is limited to one stage. - PRIMES and PRIMES 2.0. PRIMES calls for mechanisms to track model results ([de-martino-2025-primes-llm-msr-framework]). PRIMES 2.0 maps each of 23 MSR substeps to threats and mitigations ([de-martino-2025-primes2-llm-msr]). Both cover one stage. - [imran-2026-olaf-llm-annotation]. Annotation as measurement: reliability, calibration, drift, consensus, aggregation, transparency. - [ortin-2026-safellm4se]. A minimum statistical reporting standard. - [ralph-2020-acm-sigsoft-empirical-standards]. None of the 19 standards is LLM- or AI-specific, and no LLM mention was found in the four files searched (checked 2026-10-06).

Evidence synthesis (outside SE). - [flemyng-2025-ai-evidence-synthesis-position-statement] (UNCHECKED; Campbell/Cochrane joint). - The author is accountable for "the decision to use AI". - AI should be used with human oversight. - "Any use of AI or automation that makes or suggests judgements should be fully and transparently reported". - Required disclosures: name, version and date; purpose and stages affected; justification of methodological soundness; limitations. - thomas-raise-recommendations (RAISE; UNCHECKED). Assigns responsibilities to eight roles across the evidence-synthesis ecosystem. - [moghaddam-2026-arisma] (MINOR-FIX). "Every consequential scientific decision must remain human-interpretable, human-auditable, and human-accountable". It states that the PRISMA family provides no end-to-end operational standard for AI use. - [gallifant-2025-tripod-llm] (CONFIRMED). A modular reporting checklist (19 main items) that emphasises human oversight.

Integrity bodies and policy. - [ec-2026-living-guidelines-genai-research] (UNCHECKED). - Counts literature review, identifying gaps, formulating aims and developing hypotheses as "substantial use". - Asks how the tool "was used and affected the research process". - Asks for FAIR release of prompts and outputs "if relevant". - [allea-2023-european-code-of-conduct]. Asks that AI use be reported "in a way that facilitates verification or replication", and treats hiding AI use as a violation. - [blau-2024-protecting-scientific-integrity]. Asks for AI-generated inferences to be marked "with provenance information about the role of AI". - [cope-2023-authorship-ai-tools]. Disclosure goes in Methods. The content is snippet-sourced.

Venue and publisher. These cover authorship and text disclosure only: - [icse-2027-research-track-genai-policy]. Relays the ACM and IEEE policies and desk-rejects hallucinated references. - ieee-ai-generated-text-guidelines. Disclosure in the acknowledgements. - springernature-ai-guidance. Book authors disclose in the acknowledgements, introduction or preface. This governs the research write-up itself. The record flags the copy-editing disclosure rule as unresolved.

Analogues. - [kapoor-2024-reforms]: a 32-item checklist. - [mitchell-2019-model-cards]: artifact cards. - [gebru-2021-datasheets-for-datasets] (CONFIRMED): documents the collection process.

Bottom line for the gap. - Already demanded in SE: the role of the LLM, configurations, prompts, interaction and session traces, human validation of outputs, and, at "should" strength, human approval and rejection decisions in agentic studies. - Already demanded in evidence synthesis: the stages affected, a justification, and accountability for the decision to use AI. - Missing everywhere (§6): - delegated decisions as structured records (rather than raw logs or narrative) - declared authority bounds - gate semantics: what the human checked, against what criteria, and where in the lifecycle - any of the above spanning more than one stage of the empirical-SE lifecycle

4. Human-in-the-loop and provenance work

Gates and HITL inside single stages. - Literature review: - boundary cases routed to humans ([huotala-2025-aisysrev]) - prioritised review of borderline decisions plus semantic validation ([mantyla-2026-llm-screening-disagreements]) - a workflow that records disagreements and exclusion rationales ([li-2025-genai-slr-hitl], CONFIRMED) - agent trajectories made visible for live monitoring ([qiu-2025-insightagent]) - Qualitative analysis: - coding decision data (keywords, certainty) recorded ([gao-2024-collabcoder]) - human modify, reject and insert actions treated as research data ([nyaaba-2026-hacita-interpretive-authority]) - delegation levels studied as researcher preferences ([puranik-2025-assistant-not-collaborator-qda]) - a pre-analysis checklist that turns the delegation decision into a brief reviewable record, where lack of reversibility is a blocker ([prahl-2025-arc], CONFIRMED) - Question design: an accept/edit/discard step plus a Go/Pivot/Abort gate ([theodoro-2026-lri-co-assistant], CONFIRMED). The record does not say whether those decisions are logged.

Gated agentic-research architectures (mostly 2026 preprints, outside SE). - [zhu-2026-hler-human-oversight] (MINOR-FIX). LLMs reason but do not execute data work, estimation is deterministic, and three human decision gates bind the workflow. Critical failures fell from 72% to 16% (p<0.001), and an ablation suggests the gates and the deterministic computation contribute independently. - [zhu-2026-pai-econ-gated] (CONFIRMED). Blinded evaluators preferred the gated architecture in 4 of 5 tasks. Small n. - [jha-2026-paper-pilot] (CONFIRMED). Evidence-locked drafting produced zero fabricated citations. - [krebs-2026-fai2r] (UNCHECKED). Two invariants: "no parentless claim", and verification rungs only humans may grant. - [dong-2026-crossaudit] (MINOR-FIX). A git-native, cross-vendor audit loop with eight invariants. The authors are candid about limited evidence. - [zhou-2026-auditable-ai-research-writing] (MINOR-FIX). Pre-registered process observation. A frozen stopping rule halted work on a No-Go result, and a third party can recompute every primary metric. - [guo-2026-clarus] (UNCHECKED). Claims a traceable, attributable collaboration network.

Provenance infrastructure. - W3C PROV vocabulary ([w3c-2013-prov-overview]) - domain profiles: [souza-2019-prov-ml] (CONFIRMED), [padovani-2025-yprov4ml] - script-level provenance, prospective and retrospective ([pimentel-2017-noworkflow]) - agent-specific provenance: - [souza-2025-prov-agent] (CONFIRMED) relates prompts, responses and decisions to the workflow and its outcomes - [wu-2026-agenttrails] notes that chronological logs obscure dataflow - [rosen-2026-execution-lineage] treats intermediate artifacts as a replayable DAG - surveys and taxonomies: [wang-2026-agent-traces-to-trust-survey] (CONFIRMED), [dong-2024-agentops-taxonomy] - governance of agents: activity logging and agent identifiers ([chan-2024-visibility-into-ai-agents]) - the research-object approach ([binkyte-2026-airo], CONFIRMED) argues that generic disclosure does not meet auditability requirements

Oversight, sceptically. - Nominal HITL fails. Agent design impedes oversight, and extended use degrades overseers' skills ([mitchell-2026-agents-push-humans-out], CONFIRMED). More traces raise confidence, not accuracy ([grundemclaughlin-2026-overseeing-agents]). - Epistemic framing: [messeri-2024-illusions-of-understanding] (CONFIRMED) names four AI roles (Oracle, Surrogate, Quant, Arbiter) and the risk of illusions of understanding. - Institutional framing: a "Human Ownership Principle" ([vassileva-2026-aimls], CONFIRMED).

LLMs as evaluators of findings. Judges approximate humans on some tasks: Pearson 81.32 on code translation and 68.51 on code generation ([wang-2025-can-llms-replace-human-evaluators], CONFIRMED). They must be validated per task ([bavaresco-2025-llms-instead-of-human-judges]), and prompt bias can flip rankings ([zhao-2026-bias-in-the-loop-llm-judge]).

Assessment. The ingredients exist: gates, provenance models, decision records and authority constraints. They are scattered across single stages and across fields (economics, HPC/e-science, health, ML). None of them is tied to empirical-SE study types or validity threats.

5. Agentic-science systems and what they record

The "Records" column reports only what the store's records say. Where a record is silent, the cell says so.

Across systems: [ding-2026-autonomous-research-agents-verification-gap] (CONFIRMED) codes 24 runnable systems. 83% release code and 38% release seeds or execution traces, and no LLM-era system shows an externally validated in-loop oracle. [tie-2026-autoresearch-survey] (CONFIRMED) names provenance tracking and "accountable scientific closure" as open problems. [luo-2025-hidden-pitfalls-ai-scientist] (MINOR-FIX) recommends that venues mandate trace logs.

6. THE GAP

Stated honestly. The store does not support the claim that nobody records process or human interventions. It does support a narrower and defensible claim with four elements.

(a) Delegated decisions as a reportable unit. - What exists: - Many studies show that delegated choices drive conclusions: [fogliato-2026-many-ai-analysts], [baumann-2025-llm-hacking], [cummins-2025-analytic-flexibility-silicon], [hida-2026-class-imbalance-batch-screening] (CONFIRMED). - Criteria exist for whether to delegate: [ahmed-2025-llm-replace-manual-annotation-se], [calderon-2025-alternative-annotator-test] (CONFIRMED), [chew-2023-llm-assisted-content-analysis]. - Closest prior art: - [prahl-2025-arc] (CONFIRMED) makes the delegation decision a brief recorded artifact, but only for qualitative coding in health research. - [ribeiro-2026-slr-genai-graduate-course] (CONFIRMED) recommends "decision records" without specifying them. - [flemyng-2025-ai-evidence-synthesis-position-statement] holds authors accountable for "the decision to use AI" and requires reporting any use that makes or suggests judgements, but only for evidence synthesis and as disclosure. - [souza-2025-prov-agent] (CONFIRMED) captures agent decisions, but not under whose authority they were made. - How it falls short: decisions are either reconstructed afterwards ([balaji-2026-forkscope]; [fogliato-2026-many-ai-analysts] with an AI auditor) or disclosed as narrative ([ec-2026-living-guidelines-genai-research]). No ESE guidance asks authors to report which research decisions were delegated, to what, and who accepted them.

(b) Intermediate artifacts. - What exists: evidence that they matter. [wang-2025-reassessing-llm-boolean-query] shows unreported intermediate steps defeated reproduction. [kim-2026-ai-coders-discuss-disagree] shows deliberation artifacts affect accuracy. [sakana-2025-iclr-workshop-experiment-repo] (MINOR-FIX) shows a flagship release that omits them. - Closest prior art: - [dunn-2026-reproducible-llm-analyses] (CONFIRMED): "preserve LLM-generated artifacts" - [huotala-2024-llm-screening-sr-ease] (CONFIRMED): publish screening data - [ifargan-2024-data-to-paper]: result-to-data chaining - [rosen-2026-execution-lineage], [wu-2026-agenttrails]: technical lineage - How it falls short: these are per-study recommendations or system features. The ESE reporting standards ask for session traces ([baltes-2025-llm-guidelines-ese]), not a typed record of intermediate artifacts. Raw chronological logs obscure dependencies ([wu-2026-agenttrails]). The Empirical Standards are silent ([ralph-2020-acm-sigsoft-empirical-standards]).

(c) Bounded authority. - What exists: implicit constraints in systems and stage-specific guidance: - "LLMs reason but do not execute data work" ([zhu-2026-hler-human-oversight]) - "constrain LLM inference to interpretive tasks" ([jeldtoft-2026-auditable-llm-ta-workflow]) - "keep the LLM off the data path" ([dunn-2026-reproducible-llm-analyses]) - humans own the codebook ([dunivin-2025-scaling-hermeneutics], CONFIRMED) - bounded tasks with human-led inclusion ([scotti-2026-documenting-ai-not-sufficient]) - role-based responsibilities (thomas-raise-recommendations) - verification rungs only humans may grant ([krebs-2026-fai2r]) - Closest prior art: [lin-2026-llm-se-security-assurance-survey] (CONFIRMED) names "agent authority" as an assurance dimension, but for evaluating agents, not for the research process. - How it falls short: no SE or integrity source defines what an AI may decide within a study as a declared, reportable design element. The oversight literature diagnoses the absence ([mitchell-2026-agents-push-humans-out]) without supplying it.

(d) Human semantic gates as reportable process. - What exists: - Gates work: [zhu-2026-hler-human-oversight], [jha-2026-paper-pilot] (CONFIRMED), [schmidgall-2025-agent-laboratory]. - Gates exist as tool features: [huotala-2025-aisysrev], [theodoro-2026-lri-co-assistant], [mantyla-2026-llm-screening-disagreements]. - Human overrides have been recorded as data: [nyaaba-2026-hacita-interpretive-authority], [li-2025-genai-slr-hitl]. - Closest prior art: [llmguidelines-2026-session-traces] says approval and rejection decisions should be reported in agentic studies. [moghaddam-2026-arisma] makes human auditability a governing principle, for reviews only. - How it falls short: neither specifies what the human verifies, by what criteria, or where in the lifecycle gates belong. [ding-2026-autonomous-research-agents-verification-gap] finds HITL points rarely specified. [grundemclaughlin-2026-overseeing-agents] shows that unspecified oversight raises confidence without accuracy.

(e) Cross-cutting: lifecycle span and field. - Existing process governance is stage-bound: GUEST and ARISMA for reviews ([kitchenham-2026-guest-genai-slr-guidelines], [moghaddam-2026-arisma]), PRIMES for MSR ([de-martino-2025-primes2-llm-msr]), and auditable thematic analysis ([jeldtoft-2026-auditable-llm-ta-workflow]). - Whole-workflow gated provenance exists, but outside ESE and recently: mostly 2026 preprints in economics, ML and e-science (§4). - No source maps these mechanisms onto ESE study types and validity threats.

Proposed reformulation of the gap sentence.

Existing guidance already asks empirical-SE researchers to disclose models, prompts, session traces and human validation of LLM outputs, and stage-specific frameworks govern single steps such as screening or repository mining; what no SE standard yet specifies is the research process itself as a governed object — which decisions were delegated to AI and under what authority, which intermediate artifacts they produced, and which human semantic checks accepted them — recorded as reviewable provenance across the empirical lifecycle.

7. Sources that complicate the thesis

1. Guidance already goes beyond "model + prompt". - Sources: [baltes-2025-llm-guidelines-ese], [llmguidelines-2026-session-traces] (approval and rejection decisions should be reported), [flemyng-2025-ai-evidence-synthesis-position-statement] (stages affected, justification), [ec-2026-living-guidelines-genai-research] (effect on the research process; FAIR prompts and outputs), [kitchenham-2026-guest-genai-slr-guidelines]. - Must concede: the draft gap sentence is inaccurate as written. - Must argue: the step from disclosure (logs, narrative) to designed process (declared authority, typed decision records, gates with criteria). Cite these as the foundation, not as the opposition.

2. Gated, provenance-first agentic research already exists outside ESE. - Sources: [zhu-2026-hler-human-oversight], [zhu-2026-pai-econ-gated], [krebs-2026-fai2r], [dong-2026-crossaudit], [zhou-2026-auditable-ai-research-writing], [jha-2026-paper-pilot], [guo-2026-clarus], [souza-2025-prov-agent], [mo-2026-age-of-ai-agents-trustworthy-science], [luo-2025-hidden-pitfalls-ai-scientist], [bansal-2026-opendiscoverytrace], [ifargan-2024-data-to-paper]. - Must concede: gates and provenance are not new ideas. The research write-up must not claim to be first. - Must argue: novelty rests on three things: an ESE-lifecycle mapping, a connection to ESE study types and validity threats, and the six requirements as reusable design patterns. Most of these sources are 2026 preprints with small evaluations, and many are UNCHECKED, so they need verification before positioning.

3. AI matches or beats humans on several stages. - Sources:

   • screening and extraction ([cao-2025-otto-sr], MINOR-FIX)
   • annotation ([gilardi-2023-chatgpt-outperforms-crowd-workers], CONFIRMED; [tornberg-2023-chatgpt4-political-annotation]; [ahmed-2025-llm-replace-manual-annotation-se])
   • extraction at 96.3% with stable test-retest ([gartlehner-2024-claude2-data-extraction])
   • literature synthesis ([skarlinski-2024-paperqa2-superhuman-synthesis]; [asai-2026-openscholar], CONFIRMED)
   • ideation ([si-2024-llm-novel-research-ideas])
   • reanalysis ([holtdirk-2026-automated-reproducibility-assessment])
   • reproduction ([alizadeh-2026-socsci-repro-bench])
   • a human-free labelling pipeline ([senthilkumar-2026-synthcore-llm-labels])
   • Must concede: "AI cannot do this" is not the argument.
   • Must argue: governance is about accountability and inspectability, not about AI incompetence. Human gates should be risk-proportional, placed at interpretive and verdict steps. [cao-2025-otto-sr] itself relied on dual human review of discrepancies and changed significance in 3 reviews, which is a reason to record delegated decisions.

4. Output validation might be enough. - Sources: [calderon-2025-alternative-annotator-test] (CONFIRMED) and [pangakis-2023-automated-annotation-validation] offer validation against a human subset as a sufficient procedure for annotation. - Must argue: what governance adds beyond this:

   • a record of the substitution decision and its scope
   • validation where no human ground truth exists, since ground truth is itself context-limited ([hartel-2026-agentic-repository-mining])
   • the case where agreement does not equal construct validity ([pita-2026-correct-codes-wrong-reasons])

5. More traces do not make oversight effective. - Sources: [grundemclaughlin-2026-overseeing-agents] (confidence rose, accuracy did not), [mitchell-2026-agents-push-humans-out] (CONFIRMED; HITL degrades), [laux-2025-automation-bias-ai-act]. - Must concede: durable provenance alone is insufficient, and the research write-up must not promise that logging produces rigor. - Must argue: this is why gates must be semantic, with defined criteria and placement. Turn the objection into a requirement.

6. Practice lags the scenario. - Sources: [trinkenreich-2026-pulse-genai-se-research] (457 SE researchers; use is mostly writing and early-stage work, and methodological tasks stay human-driven). [gomes-2026-llm-assisted-ese-slr] finds integration automation-oriented and concentrated in data processing. - Must concede: fully agentic ESE research is still emerging. - Must argue: anticipatory design, using concrete 2026 SE instances: [hartel-2026-agentic-repository-mining], [nyyssola-2026-llm-rapid-reviews-tool-discovery] (about 4 human-hours for a full rapid review), [agarwal-2026-3100-opinions-code-review] (CONFIRMED; an LLM-assisted coding pipeline shipped as a method), [wehr-2026-closing-empirical-loop].

7. For reflexive qualitative work, AI participation is contested. - Sources: [jowsey-2025-reject-genai-reflexive] (419 signatories). The rebuttal is [ibrahim-2026-should-genai-reflexive]. [karhu-2026-vibe-research-qualitative] distinguishes small-q from Big-Q research. - Must do: scope the framework to small-q, coding-reliability, MSR and quantitative work, or treat the decision not to delegate as a legitimate gate outcome.

8. The risk is not uniform. - Sources: agents' estimates stay aligned with human consensus, while their verdicts flip under framing ([alizadeh-2026-coding-agents-social-science]). Agents refuse overt p-hacking ([asher-2026-claude-code-codex-p-hack], low confidence and snippet-based: not load-bearing). SE survey quantitative results stayed stable after AI-response filtering ([desouzasantos-2026-fraudulent-ai-se-surveys]). Average contamination is low ([zhou-2025-lessleak-bench]). - Must argue: concentrate gates where the risk concentrates, at interpretation and verdicts. Avoid alarmism.

9. Preregistration is a competing remedy. - Sources: [thomas-2026-preregister-next-llm] (MINOR-FIX), [vaccaro-2026-preregistration-ai-agents], [bisht-2026-agentic-ai-scientists-not-built] (MINOR-FIX). - Must argue: preregistration fixes intent before a run but does not record what the agent did. It complements provenance; it is not a substitute. The research write-up should incorporate it, perhaps under executable reproducibility or bounded authority.

10. A judge-replacement agenda points the other way.

    • Source: [he-2025-llm-as-judge-se-roadmap] envisions LLM judges as human surrogates by 2030.
    • Must argue: against blanket automation of evaluation, citing [zhao-2026-bias-in-the-loop-llm-judge] and [bavaresco-2025-llms-instead-of-human-judges]. Concede that the evidence is task-dependent ([wang-2025-can-llms-replace-human-evaluators], CONFIRMED).