Automated Data Extraction for Systematic Reviews: Reclaiming the Most Expensive Hours Without Losing the Audit Trail

The systematic review is the most respected object in clinical evidence. It sits at the top of the evidence hierarchy, it anchors every Clinical Evaluation Report (CER) under EU MDR, and it is the format Notified Bodies, HTA agencies, and guideline panels trust above any single trial. The methodology is mature, the reporting standards are excellent, and the discipline of the field is real.

However, buried inside that respected object is a step almost nobody defends on its merits and almost everybody performs the same way they did twenty years ago: a human opens a PDF, hunts through the tables and the methods section, and types numbers into a spreadsheet. Then, if the review is done properly, a second human does the entire thing again. This is data extraction, and it is simultaneously the most labor-intensive step of a review, one of the most error-prone, and the one most ready to be automated.

The thesis of this post is straightforward. Manual data extraction is a bottleneck we tolerate out of habit, not necessity. The reason we have been slow to automate it is not that the reading is impossible for a machine — modern language models read full-text papers well — but that the old automation attempts could not prove where a number came from, and provenance is the one thing a regulatory review cannot give up. Solve the provenance problem, and the case for automating extraction becomes overwhelming.

In short:

Data extraction routinely consumes a large share of the months a systematic review takes, and dual independent extraction — the correct method — doubles that reading labor.
Manual extraction is not merely slow; it is a documented source of errors serious enough to change a pooled result.
The established tools (Rayyan, Covidence, DistillerSR, and others) mostly automate the screening funnel and the management of manual extraction. The reading of the full text is still, in the mainstream workflow, a human task.
The right target for automation is not the keystroke. It is the reading, matched to source evidence, with every extracted value bound to the document, protocol, model, and quote it came from.
The tool described here does exactly that, and it is arriving as a free, restricted demo on this site.
On our internal benchmark — 549 extracted value pairs across four clinical corpora — the pipeline reaches 94.8% precision and 91.6% recall with zero hallucinated values, and it reports its numeric-field weak spot (88.2%/78.9%) honestly rather than hiding it.

The true cost of the most manual step

The case against manual extraction is not aesthetic. It is quantitative. Consider the four costs it imposes.

Time. A systematic review of medical interventions takes, on average, around 67 weeks from registration to publication and involves roughly five people (Borah et al., 2017). Extraction is not the whole of that, but it is a substantial and irreducible fraction, because it scales linearly with the number of included studies and the number of parameters per study. A review of 30 trials extracting 20 fields each is 600 values to locate, read, transcribe, and check. Twice, if done to standard.

Money. The time converts into cost. Published estimates put the price of a single systematic review upward of USD 100,000 in staff effort (Michelson & Reuter, 2019). For a small or mid-size medical device manufacturer running a CER or a PMCF evidence update, that figure lands squarely on the critical path to market.

People. Cochrane and every serious methods guide recommend dual independent extraction: two reviewers extract the same data separately, then reconcile. This is not bureaucratic caution. It exists because single extraction demonstrably produces more errors than double extraction (Buscemi et al., 2006). The correct method is, by construction, the one that doubles the most tedious labor in the project.

Error. Even with care, manual extraction is leaky. Errors in data extraction are frequent enough that a methodological review found them to be a routine quality problem rather than an occasional slip (Mathes et al., 2017), and, in a classic audit of meta-analyses using standardized mean differences, extraction errors were common enough that correcting them changed the pooled estimate in a meaningful minority of cases (Gøtzsche et al., 2007). The uncomfortable implication is that some published pooled effects are artifacts of a mis-typed number, not of the underlying evidence.

Put plainly: the step we automate least is the step that costs the most and fails the most. That is exactly backwards.

What today's tools automate, and what they quietly leave to you

The systematic review software market is not empty. It is genuinely good at what it does. But it is worth being precise about which step each tool automates, because the marketing language ("AI-powered", "accelerate your review") blurs a distinction that matters enormously.

Tool	What it primarily automates	What it leaves to the human
Rayyan	Deduplication and title/abstract screening, with machine-learning prioritization of the reference list (Ouzzani et al., 2016).	The entire full-text data extraction step.
Covidence	Screening workflow plus a structured extraction template with dual-reviewer reconciliation.	Reading each PDF and typing every value into the template.
DistillerSR	Enterprise workflow, configurable forms, audit logging, and AI assists that mostly prioritize and classify references.	The substantive reading of the full text into fields.
EPPI-Reviewer / SRDR+ / RevMan	Review management, coding, and synthesis/meta-analysis.	Manual data entry from source documents.

The pattern is consistent. These tools are excellent at the screening funnel — turning 4,000 hits into 40 includable studies — and at managing the extraction: giving you a form, enforcing dual entry, tracking disagreements, and logging who did what. That management is valuable, and none of it is what I am criticizing.

What they largely do not do is read the paper for you. In the mainstream workflow, the number that ends up in the "median progression-free survival" cell of your extraction form got there because a person read Table 2, found the value, and typed it. The software structured and reconciled that human labor. It did not remove it.

This is not a knock on the vendors. It is a statement about where the frontier sat until recently. Reading a heterogeneous clinical PDF — with its two-column layout, its split tables, its footnoted subgroups — and pulling the right value into the right field is genuinely hard, and the earlier generation of rule-based extractors were too brittle to trust. Some tools are now adding large language model features, and that is the right direction. The question is no longer whether a machine can read the paper. It is whether it can do so in a way a Notified Body will accept.

The reframing: extraction is a provenance problem, not a typing problem

Here is the inversion at the center of this argument. We have always described data extraction as a transcription task — move the number from the PDF to the spreadsheet. That framing is why automation felt reckless. If extraction is just typing, then an AI that types is an AI that hallucinates numbers into your evidence base, and no serious reviewer would allow it near a CER.

But transcription was never the hard or the valuable part of extraction. The valuable part is the judgment and the traceability: finding the correct value among several candidates, in the correct study arm, and being able to point afterwards to the exact sentence or table cell that justifies it. In a regulatory systematic review, a number without its source is worthless, because an assessor can neither verify it nor reproduce it.

So the correct design goal is not "an AI that extracts data." It is an AI that extracts data and hands you the receipt — the quote, the section, the page, and a machine-checkable trail binding that value to the document, the protocol, the model, and the run that produced it. Automate the reading, and refuse to give up the audit trail. Do that, and automation stops being a risk to rigor and becomes a way to increase it, because a machine that must ground every value to a source quote is, in one respect, more disciplined than a tired human at hour six of a charting session.

A different approach: auditable automation

This is the design principle behind the data extraction platform I have been building, and which is now arriving as a demo on this site. It turns a PICO-scoped batch of full-text PDFs into a precise, traceable extraction bundle in which every exported value links back to a document, an extraction run, a protocol, a segmentation snapshot, a model, and a source quote. Provenance is not a feature bolted on at the end. It is the architecture.

Three properties define it.

It is deterministic-first, and the language model is on a short leash. The pipeline does as much as possible with deterministic code — parsing, sectioning, metadata handling — and calls a frontier model only for the genuinely interpretive step of finding a field's value in the relevant section. Bibliographic fields such as DOI, PMID, journal, and year never reach the model at all; they are resolved from fetched metadata. The model is asked to read, not to invent.

It grounds every value and reports confidence as components, not as one vague number. When the model returns a value, a grounding step matches that value back to the source text and records the quote, the section, and the page. Each extracted value then carries four separate signals: whether the value was present, how well it grounds to the source, whether it passes domain validation, and an overall score — never blended into a single reassuring percentage that hides its own weakness. Numeric grounding is deliberately conservative: a bare "42" is not allowed to match inside "0.42", and a number is only treated as high-confidence when it is anchored by surrounding section or quote context. Values that clear the thresholds auto-accept; everything else is flagged for a human. The human stays in the loop by design, but reviews a triaged shortlist instead of re-reading every paper.

Its precision is measurable — and so is its restraint. In our own testing against four clinical corpora — breast and colorectal fluorescence-imaging studies and a shoulder-arthroplasty meta-analysis, 549 extracted value pairs in total, scored against the meta-analyses' own published tables as ground truth — the pipeline reaches 94.8% precision and 91.6% recall. More telling than either number: across all 549 checked pairs it produced zero hallucinated values — not one invented number or fact that was not in the source. The weak stratum is the one the whole field struggles with: on purely numeric fields precision falls to 88.2% and recall to 78.9%, in line with independent 2025 reports that numeric outcomes are where LLM extraction is least reliable. Upstream, section attribution during parsing scores in the mid-90s across 40 documents. These are first-hand measurements on small, named corpora, reported with precision and recall kept separate rather than blended into one reassuring percentage — and deliberately not a head-to-head against any other tool, whose numbers rest on a different corpus, a different ground truth, and a different scoring method. None of them removes the human review gate. What they show is that the machine can now do the reading and the grounding while the human does the checking — and that when the machine is unsure, it says so instead of inventing.

Under the hood, briefly

The internal workflow is an eight-step pipeline, but the shape is easy to describe.

Define the question and upload the papers. A PICO question plus a batch of full-text PDFs.
Specify the parameters. The fields to extract, with data types and export column names. The system can suggest a starting set from the question alone.
Generate an extraction protocol. A confirmed, hashed JSON artifact that fixes what "correctly extracted" means for this review.
Parse and segment. Each PDF is parsed with PyMuPDF into a coordinate-preserving block model and sectioned into IMRaD structure, producing an immutable, content-hashed snapshot.
Extract with grounding. The frontier model extracts each parameter from the relevant section; grounding and the four confidence signals are computed in the same run.
Review and correct. Low-confidence and ungrounded values are surfaced for human judgment. Corrections are append-only — the original is never overwritten, the latest correction wins in the export, and both stay auditable.
Export the bundle. The deliverable is built server-side from durable database state: a wide results table (CSV and XLSX), a machine-readable evidence.json of provenance per value, a corrections log, and a MANIFEST.sha256 that binds every file in the bundle to the protocol hash, the snapshot hash, the run ID, the model ID, and the prompt version.

That manifest is the point. It means the extraction is not just fast; it is reproducible and inspectable. An assessor can trace any cell in your results table back to the sentence in the paper that justifies it, and confirm that the deliverable was built from exactly the inputs it claims.

The stack is FastAPI with async SQLAlchemy, PyMuPDF, and the Anthropic SDK on the backend, and a React frontend, running on Postgres. In line with the rest of MH-Analytics, it is built to run on EU infrastructure, with the demo hosted on secure servers in Germany.

Try it

A free, restricted public demo is live on this site. The user-facing flow is three steps: define a research question and approve the suggested extraction parameters, upload up to three full-text PDFs, and extract — watch the pipeline run and review the results table with its confidence pills and expandable source quotes.

Privacy is built into the demo, not promised around it. Uploaded files are processed ephemerally, deleted after processing, hosted in the EU, and never used to train anything.

Try the live demo, or get in touch to arrange a walkthrough on your own papers.

What to demand from any extraction tool

If there is one thing to take from this post, it is a checklist. Automating extraction is not the risk. Automating it without provenance is. So whether you evaluate the tool or anyone else's, demand the following, and treat their absence as disqualifying:

The value and its evidence travel together. Every extracted number should arrive with the quote, section, and page it came from. A value you cannot trace is a value you cannot defend.
Confidence is reported as components, not one number. Presence, grounding, and validation are different questions. A single blended percentage hides which one failed.
The deliverable is cryptographically bound to its inputs. A manifest hashing the protocol, the source snapshot, the model, and the run turns "trust me" into "check it yourself."
Corrections are append-only. Human overrides must not silently erase the machine's original output. Both belong in the audit trail.
The human stays in the loop, on the hard cases. Automation should triage the reviewer's attention to the uncertain values, not replace the reviewer.
The data stays where your regulator expects it. For EU device work, that means EU hosting and no quiet reuse of your documents.

This is not a wish list aimed at any one vendor. It is the specification that lets a Notified Body accept automated extraction at all, and it is entirely achievable today.

Conclusion

The systematic review is a rigorous instrument with one stubbornly manual joint. We have automated the search, we have automated screening, we have automated the meta-analysis and the forest plot. The one step left almost entirely to human hands is the reading of the full text into structured data, and it is the step that costs the most weeks, the most money, and the most errors.

The reason we hesitated to automate it was sound: an AI that types numbers into your evidence base without proof is a liability, not a tool. But that hesitation confused the task. Extraction was never really about typing. It was about finding the right value and proving where it came from, and a system that grounds every value to a source quote and binds every deliverable to its inputs does the second part better than a tired human ever could.

So the conclusion stacks. Manual extraction is not only the slowest and most error-prone step of a systematic review. It is also the most automatable, and automating it correctly does not weaken the audit trail your CER depends on — it strengthens it. The tools we already use took the drudgery out of screening. It is time they took it out of extraction too, and did so without asking anyone to trust a number they cannot trace.

Reclaim the hours. Keep the receipts.

References

Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017). Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open, 7(2), e012545.
Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., & Klassen, T. P. (2006). Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology, 59(7), 697–703.
Gøtzsche, P. C., Hróbjartsson, A., Marić, K., & Tendal, B. (2007). Data extraction errors in meta-analyses that use standardized mean differences. JAMA, 298(4), 430–437.
Mathes, T., Klaßen, P., & Pieper, D. (2017). Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Medical Research Methodology, 17(1), 152.
Michelson, M., & Reuter, K. (2019). The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials. Contemporary Clinical Trials Communications, 16, 100443.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan — a web and mobile app for systematic reviews. Systematic Reviews, 5(1), 210.

Dr. Marc Harms MH-Analytics