← Governed Agentic Empirical Research
Does the tooling cover the work?
A completeness benchmark for agentic-research · planned study, no results yet
agentic-research grew one fix at a time, each prompted by something that went wrong on a real paper. That is a good way to build a tool and a poor way to know what it misses. This study asks the question directly: set against what doing research and writing a paper actually involves, what does the tool support, what does it only gesture at, and what does it leave entirely to the researcher?
Why existing benchmarks don't answer this
Benchmarks for research agents mostly score the output of an autonomous run.PaperBench has agents replicate ICML papers from scratch, graded against rubrics written with the original authors.ResearchClawBench hides a published paper and asks agents to re-discover it from its literature and raw data.ARAC-Bench comes closest in spirit. It scores how completely an autonomous system's trajectory covers expected methodology across proposal, experiment and synthesis, and finds the experiment stage is where systems fall down.AARRI-Bench runs tasks across several agent harnesses and finds that the model matters far more than the harness. On the writing side,paper-reconstruction evaluation checks AI-written papers for presentation quality and hallucination, anda multi-model review study asks whether AI reviewers can rank AI-scientist systems at all.
All of these treat the agent as the researcher. agentic-research is built on the opposite assumption: a human does the research, and the tool scaffolds, checks and gates the work around them. For a tool like that, the useful questions are different. Does it cover the lifecycle? Do its gates actually close? Where does it hand a decision back to the human, and does it say so?
Research questions
- Coverage. Which activities of the research and paper-writing lifecycle does the tool support, and how strongly?
- Enforcement. Do its mechanical checks catch the defects they claim to catch, including slightly disguised ones?
- The human role. At which activities does the tool hand judgment to a person, and is that hand-off explicit in the tool or left implicit?
How it will work
Stage 1: a coverage audit
The reference is a lifecycle model built from published sources, with every activity traced back to the item it came from: the general standard of theACM SIGSOFT Empirical Standards, the 14 contributor roles of CRediT (ANSI/NISO Z39.104-2022), the four research stages of theLLM4SR survey, the six-stage workflow of arecent survey of AI scientists, and ARAC-Bench's stage rubric. A script inventories every skill, agent, command, check and template in a pinned version of the plugin. Two people then code each activity independently on a four-step scale:
| Level | Meaning |
|---|---|
| Enforced | A mechanical check fails when the activity isn't done. |
| Procedural | An agent or skill procedure covers it, but nothing fails if it is skipped. |
| Scaffolded | A template or folder exists for it, with no procedure behind it. |
| Absent | Nothing in the tool addresses it. |
Agreement between the coders is reported before they reconcile. An agent's proposed coding can be compared against theirs, but it is never used as the answer key.
Stage 2: do the gates close?
For every claim the plugin makes about what its checker catches, a script injects that defect into a clean fixture paper and records whether the checker fails. Each fault also gets disguised variants, such as a citation command with options or a placeholder inside a comment. Clean fixtures run alongside as controls that must pass. No model is involved, so every run is deterministic and repeatable from one command. Runs where an agent performs the plugin's procedures, for example verifying a reference list that has fabricated entries mixed in, are designed and pre-registered now. They will run only once thresholds and a budget are set.
The tool is also a subject
The study's own repository is being set up by OpenCode, a different agent harness, following the plugin's procedures as written for Claude Code. A step-by-step log of what it could follow, what it had to adapt and what it could not do is a direct test of the harness-independence gap.
Where people decide
- The lifecycle reference model is approved by a person before anything is scored against it.
- Coverage coding is done by two people. An agent's version is a comparison, never the gold standard.
- Pass thresholds and the budget for agent runs are set by a person and committed before those runs happen.
- Every change to the study repository goes through governance: draft PRs, checks, human merge.
Status
Planned. The design and tooling are being built in a private repository as of 9 November 2026, and a workshop paper is in preparation. No result on this page should be read as a finding until it appears here with its data.