Consensus Searches

The query asked about using AI to formulate research questions and design empirical SE studies. Almost nothing in the ranked list addresses that stage directly; the results instead cluster around (a) broad LLM4SE surveys, (b) methodological guidance for studies that involve LLMs, and (c) reproducibility audits. The closest to the design stage is [liang-2024-gpt4-replicate-ese]: 14 SE researchers judged GPT-4-generated assumptions and analysis plans for seven empirical papers; GPT-4 surfaced correct assumptions but missed common knowledge about SE data, and its code had many small implementation errors. [liang-2024-llms-emse-tasks-emsebench] similarly benchmarks LLMs on human-participation scenarios from published EMSE studies.

The guidance thread is the strongest: [wagner-2025-evaluation-guidelines-llm-studies] proposed the first holistic guidelines, and [baltes-2025-llm-guidelines-ese] (22 authors) consolidates eight guidelines, including reporting session traces/interaction logs and validating LLM outputs against human judgment. The anchor [gomes-2026-llm-assisted-ese-slr] (50 primary studies) finds LLM use concentrated in data processing and analysis, automation-oriented, with limited decision support and incomplete reporting. Reproducibility audits are sobering: [angermeir-2025-reproducibility-commercial-llm-ese] could fully reproduce none of five executable ICSE/ASE 2024 studies, and [siddiq-2025-llm4se-reproducibility-crisis] finds persistent "reproducibility smells" across 640 papers. [williams-2025-empirical-sustainability-llm-se-reflection] raises benchmarking, contamination and sustainability concerns. Qualitative-method entries ([martinez-montes-2025-llm-thematic-analysis-se], [bano-2023-llms-qualitative-se]) also surfaced here. Background: [hou-2024-llm4se-slr].

Gap signal: no result treats AI-assisted question formulation or study design as a reportable, governed decision; the stage is essentially empty in this channel.

This is the most mature stage in the channel, but the evidence base is overwhelmingly biomedical; only two SE-specific primary studies appeared. In SE, [thode-2025-llm-selection-phase-slr] reports recall up to 98% but precision around 27% for LLM-based study selection on two published SE reviews, recommending a two-LLM approach, and [huotala-2024-llm-screening-sr-ease] finds some LLM/prompt combinations match human screeners but are not significantly more accurate, and asks future reviews to publish screening data.

Outside SE the picture is mixed. A systematic review of 63 studies ([laignelot-2026-llm-systematic-review-sr]) reports median agreement around 0.9 for screening but much lower and more variable figures for risk-of-bias assessment. [khraisha-2024-gpt4-systematic-review-screening] shows apparent human-level accuracy dropping once chance agreement and class imbalance are corrected. [gartlehner-2024-claude2-data-extraction] reports 96.3% extraction accuracy with high test-retest reliability — evidence that complicates the claim that LLM steps are inherently unstable — while [oami-2026-llm-data-extraction-prompting] reports outcome-extraction inter-session consistency of only 44.8–65.6%. [dennstadt-2024-llm-title-abstract-screening] shows small prompt or scale changes materially shift performance. [li-2025-genai-slr-hitl] builds an explicit human-in-the-loop design that logs LLM–human disagreements and their rationales to refine criteria. [scherbakov-2025-llm-literature-reviews-jamia] finds only 26 of 172 included works were reviews that acknowledged LLM use — a disclosure gap. The anchor [gomes-2026-llm-assisted-ese-slr] and background [hou-2024-llm4se-slr] also ranked.

Gap signal: results measure screening/extraction accuracy; almost none record which inclusion decisions were delegated, how disagreements were adjudicated, or keep intermediate decisions as auditable artifacts ([li-2025-genai-slr-hitl] is the partial exception).

Roughly half the list is noise (code-generation and repository-level coding papers that match "repository" lexically). The relevant half is coherent and methodologically pointed. [ahmed-2025-llm-replace-manual-annotation-se] (MSR 2025) finds that replacing some human annotation with LLMs on ten SE annotation tasks can yield inter-rater agreement equal or close to human agreement, and proposes model-model agreement and model confidence to decide when substitution is safe — a result that complicates any blanket claim that LLM annotation is unreliable. [pangakis-2023-automated-annotation-validation] (27 social-science annotation tasks) argues the opposite emphasis: performance is task- and dataset-contingent, so every LLM annotation process must be validated against human labels. [tan-2024-llm-data-annotation-survey] surveys LLM-based annotation and its assessment.

SE-specific method frameworks appear: [de-martino-2025-primes-llm-msr-framework] (PRIMES) packages a checklist for prompt refinement and model comparison and stresses mechanisms for tracking model results, and [imran-2026-olaf-llm-annotation] (OLAF) argues annotation should be treated as a measurement process with reliability, calibration, drift, and transparency constructs. [romero-arjona-2026-lms-in-msr-survey] (177 papers) examines adoption and the availability of supplementary materials. Outside SE, [rogic-2026-llm-annotation-genomics-curation] asks the model for supporting verbatim quotes so curators can verify each annotation — a lightweight provenance mechanism. [siddiq-2025-llm4se-reproducibility-crisis] and [hou-2024-llm4se-slr] also ranked.

Gap signal: validation-against-humans is well argued; recording which labels were delegated, model drift over the labelling run, and per-item evidence trails is proposed ([imran-2026-olaf-llm-annotation], [rogic-2026-llm-annotation-genomics-curation]) but not standard.

Four SE-specific papers form a small but active cluster, three of them from the WSESE methodological workshop or adjacent venues. [leca-2025-llm-qualitative-analysis-ese-mapping] maps 20 studies and finds LLMs used for coding, thematic analysis and categorisation, with output variability, loss of nuance, and privacy/transparency concerns. [martinez-montes-2025-llm-thematic-analysis-se] operationalises Braun and Clarke's reflexive TA with documented prompts: expert evaluators preferred LLM-generated codes 61% of the time, yet LLMs fragmented data and missed latent interpretations. [alves-2026-llm-thematic-analysis-isern-workshop] (25 ISERN researchers) frames LLMs as support, not substitute, and calls for continuous human oversight. [alshaikh-2026-prompt-llm-qual-coding-psych-safety] runs ten repeated runs per configuration and reports intra-model variance and systematic label over-prediction (bias ratios up to 5.25x).

Outside SE, agreement numbers vary widely: [zhang-2025-qualigpt-irr] reports Cohen's kappa 0.57 (inductive) and Fleiss' kappa 0.46 (deductive), while [jain-2025-multi-llm-thematic-analysis-kappa] reports run-to-run kappa above 0.80 for three models — a figure about self-consistency that could be read as evidence that LLM coding is already reliable, complicating the thesis. [dunivin-2025-scaling-hermeneutics] describes a hybrid workflow that keeps codebook development human-led and adds chain-of-thought justifications, and [dai-2023-llm-in-the-loop-ta] proposes human–LLM collaborative TA with similar coding quality to human coders.

Gap signal: reliability is reported as an agreement statistic; none of these results records the codebook revisions, prompts per iteration, or which interpretive decisions the human retained as a reportable artefact.

No SE-specific analysis study surfaced, but the list is strongly on-thesis. Two findings frame the stage. First, LLM analysis is a distribution, not a value: [cui-2026-same-prompt-different-outcomes] ran 480 attempts (six models, two prompting strategies, four temperatures, ten runs each) and found considerable variation in results under identical configurations, recommending multiple independent runs. [fogliato-2026-many-ai-analysts] (PNAS) shows autonomous AI analysts reproduce many-analyst dispersion in effect sizes and conclusions, that outcomes are steerable by persona or model choice, and argues for multiverse-style reporting plus full prompt disclosure. Second, structure helps: [zeng-2025-airepr] finds workflows with higher reproducibility also yield more accurate analyses across 1,032 tasks; [lee-2026-autometa] enforces an explicit, auditable protocol with inter-agent critique and a deterministic statistics module, and ablation shows removing critique or protocol enforcement degrades accuracy.

Agents as reproducers/auditors: [kohler-2026-agentic-reproduction-social-science] reproduces 48 papers from methods text alone under information isolation and attributes failures partly to underspecified papers; [holtdirk-2026-automated-reproducibility-assessment] (N=180) matches the original qualitative conclusion in 80% of cases; [baek-2026-artisan-table-reproduction] frames reproduction as producing an auditable re-runnable script and found 20 new errors across 23 SE papers. [ifargan-2024-data-to-paper] programmatically back-traces information flow from data to paper — evidence that some systems already record provenance, which complicates a "nobody records anything" framing. [nguyen-2026-replicatorbench] evaluates the replication process, not just outcomes.

Gap signal: analytic choices are shown to drive results, yet only [ifargan-2024-data-to-paper] and [lee-2026-autometa] treat the decision trail as a first-class, inspectable record.

The AI-scientist literature is large and fast-moving; nothing here is SE-specific. Capability claims are strong: [yamada-2026-ai-scientist-nature] reports an end-to-end pipeline whose generated manuscript passed first-round review at an ML workshop (70% acceptance rate), building on [lu-2024-ai-scientist]; [mitchener-2025-kosmos] cites every report statement to code or literature, and independent scientists judged 79.4% of statements accurate; [wehr-2026-closing-empirical-loop] ran three psychology studies with 288 human participants end to end. The survey [wei-2025-agentic-science-survey] frames this as "Agentic Science".

Evaluation and replication benchmarks are sobering. [starace-2025-paperbench]: best agent replication score 21.0% on 20 ICML papers; [siegel-2024-core-bench]: 21% on the hardest computational-reproducibility tasks; [gaddipati-2026-mlreplicate]: human reviewers found methodological flaws and hallucinated results across six systems, and 59% of accepted automated reviews contained fabricated or unsupported claims; [agrawal-2026-can-ai-conduct-autonomous-research]: none of eight frameworks completed a full research cycle, and every one produced sophisticated hallucinations. [ding-2026-autonomous-research-agents-verification-gap] codes 24 runnable systems: 83% release code but only 38% release seeds or execution traces, and no LLM-era system demonstrates an externally validated in-loop oracle. [bisht-2026-agentic-ai-scientists-not-built] recommends a centralized preregistration repository for AI-generated hypotheses. [nguyen-2026-replicatorbench] (also q5) evaluates the replication process stage by stage.

Complicating evidence: [mitchener-2025-kosmos] already makes reasoning traceable to code/literature — some systems do log a lot. Gap signal: what is logged is the agent's trace; delegated authority, human interventions, and gate decisions are not reported ([ding-2026-autonomous-research-agents-verification-gap] codes HITL points as largely undocumented).

This is where the SE community has done the most explicit work. [baltes-2025-llm-guidelines-ese] is the reference point: a taxonomy of seven study types and eight guidelines with must/should levels, including declaring LLM usage and role, reporting model versions and configurations, documenting system and prompt design, reporting session traces (interaction logs and runtime traces), and validating outputs against human judgment; it builds on [wagner-2025-evaluation-guidelines-llm-studies]. [korn-2026-reporting-llm-prompting-ase] (nearly 300 papers, 105 PC-member survey) finds misalignment between practice and reviewer expectations on version disclosure, prompt justification and threats to validity. Audits quantify the problem: [angermeir-2025-reproducibility-commercial-llm-ese], [siddiq-2025-llm4se-reproducibility-crisis] (proposes a Reproducibility Maturity Model), and [williams-2025-empirical-sustainability-llm-se-reflection].

Stage-specific frameworks: [de-martino-2025-primes2-llm-msr] (PRIMES 2.0: nine threats, 25 mitigations, 23 substeps for LLM-based MSR) extending [de-martino-2025-primes-llm-msr-framework]; [martinez-montes-2025-llm-thematic-analysis-se] and [leca-2025-llm-qualitative-analysis-ese-mapping] for qualitative work. Outside SE, [gallifant-2025-tripod-llm] (19 items, 50 subitems, Delphi) emphasises human oversight, and [dunn-2026-reproducible-llm-analyses] makes the useful distinction of whether the LLM sits on or off the data path. [lin-2026-llm-se-security-assurance-survey] derives a minimum reporting protocol that explicitly separates evidence provenance and agent authority and names under-reported human intervention as a validity threat. Anchor [gomes-2026-llm-assisted-ese-slr] and [hou-2024-llm4se-slr], [bano-2023-llms-qualitative-se] also ranked.

Complicating evidence: [baltes-2025-llm-guidelines-ese] already asks for interaction logs and human validation — more than "model + prompt". Gap signal: none of the guidelines asks authors to report the allocation of authority (which decisions were delegated, which gates a human held) as a process artifact; [lin-2026-llm-se-security-assurance-survey] comes closest, for agent evaluation rather than research process.

This query surfaced the closest neighbours to the research's thesis — and a warning that the space is filling quickly, mostly through 2026 arXiv preprints outside SE. Provenance models: [souza-2025-prov-agent] (PROV-AGENT) extends W3C PROV to capture prompts, responses and agent decisions within end-to-end workflow provenance; [krebs-2026-fai2r] packages a PROV-O extension as an executable agent skill with CI gating and "verification rungs that only humans may grant"; [binkyte-2026-airo] treats AI interactions as inspectable Research Objects with interaction logs and metadata packaging.

Gated human-in-the-loop architectures with empirical evidence: [zhu-2026-hler-human-oversight] reports a pre-specified 2×4 factorial experiment (280 runs) where an unconstrained multi-agent baseline failed critically in 72% of runs versus 16% with three binding human decision gates and deterministic data work; [zhu-2026-pai-econ-gated] uses inspectable intermediate records and human checkpoints for irreversible decisions; [jha-2026-paper-pilot] defines eight approval gates and evidence-locked revision, eliminating fabricated citations in a controlled benchmark. Audit infrastructure: [dong-2026-crossaudit] (git-native cross-vendor audit, human-written versioned rulebook) and [zhou-2026-auditable-ai-research-writing] (hash-bound provenance, red-line gates, pre-registered process observation). Guidelines/governance: [moghaddam-2026-arisma] (every consequential decision human-interpretable, human-auditable, human-accountable), [prahl-2025-arc] (pre-analysis HITL posture for LLM-assisted coding), and [vassileva-2026-aimls] (AI Methodology Logging Standard). Surveys: [tie-2026-autoresearch-survey] names provenance tracking and accountable closure as open problems; [ding-2026-autonomous-research-agents-verification-gap] (also q6).

Complicating evidence: several 2026 preprints already propose gated, provenance-first research workflows ([zhu-2026-hler-human-oversight], [jha-2026-paper-pilot], [dong-2026-crossaudit], [krebs-2026-fai2r]). The research write-up's novelty must therefore rest on the empirical-SE framing and the requirement set, not on the idea of gates and provenance per se. None of these is SE-specific.