AI for document review: the layers you can hand to an agent, and the ones you cannot

Document review never appears as a line item and never stops consuming people. It sits inside diligence rooms, discovery sets, claims files, onboarding packs, credit files and regulatory inspections. The work is the same underneath: read a pile of documents, work out what each one is, pull out the facts that matter, check them against a standard, and decide what needs a person's attention.

That shape makes it a good candidate for AI agents, and also dangerous to automate carelessly. The reading is mechanical. The consequences are not.

What document review actually costs you today

Ask a team how long review takes and you get the per-document number: eight minutes for a policy file, two hours for a complex claim. That is usually the smallest part of the cost.

The larger costs sit around it. Queue time, where documents wait days for a reviewer who is booked. Rework, where a file comes back because a schedule was missing or a value came off the wrong page. Inconsistency, where two reviewers reach different conclusions on similar files and nobody notices until an audit does. The seniority tax, where a specialist spends the first half of every file locating information rather than judging it. And the ceiling: volume you decline, samples you shrink, portfolio questions you never ask.

An agent earns its place here not by being cleverer than a reviewer, but by removing the waiting, the locating and the transcribing so the reviewer starts at the judgement.

Intake and classification

Every review system starts with a mess at the front door. Documents arrive by email, portal, shared drive and scanner, as native files, exports, photographs of paper and PDFs of PDFs. Half are not named usefully.

An intake agent normalises formats, runs text recognition on anything scanned, and flags pages where the text layer is too poor to rely on rather than quietly guessing. It splits combined files into component documents, which matters more than people expect, because a sixty page upload is frequently nine documents stapled together.

It then classifies. Each document gets a type, a party, a date, and a link to the matter, case or entity it belongs to. From there it applies the routing rules you already use informally: this class goes to the standard lane, this one needs a specialist, this one duplicates something already reviewed, this one is a near duplicate with three changed clauses.

Two more intake jobs pay for themselves quickly: completeness checking, which chases missing items before a reviewer opens anything, and deduplication, which in discovery and diligence sets removes a meaningful share of the volume outright.

Extraction and comparison against a standard

This is the core of the work. Once a document is identified, the agent extracts the fields that matter for its type and checks them against whatever standard governs the review. The standard varies and the mechanics do not. In diligence it is the buyer's checklist and the definition of a red flag. In regulated file review it is the policy the file must evidence compliance with. In discovery it is relevance, responsiveness and privilege. In supplier or credit review it is your acceptance criteria.

The output for each document should be a structured record with three parts. What the document says, quoted verbatim, with a citation to the page and clause. What the standard requires. And the gap between them, classified as compliant, non compliant, or unresolved. That third state is the most important design decision in the system. An agent that must always produce an answer will produce a confident wrong one. An agent permitted to say "I could not determine this" turns an accuracy problem into a routing problem, which is far easier to manage.

Well built extraction also checks for absence. The expensive failure is rarely a wrong value, it is a missing document, an unsigned page, an expired certificate or a clause that should have been there and was not.

Exception and escalation routing

Once documents carry structured findings, routing becomes straightforward, and this is where most of the throughput gain arrives. Clean files with all fields located, all checks passing and high confidence go to a light confirmation step. Files with a few routine deviations go to the standard queue with the issues listed and the relevant text pinned. Files with unresolved fields, poor source quality, contradictions between documents, or findings above a materiality threshold go to a named specialist with a note on what the agent could not settle and why.

Some escalations should be unconditional regardless of confidence: anything touching a regulated decision, anything above a value threshold, anything involving a party on a watch list, anything where the finding would change an outcome for a person. Write those down as rules, not confidence scores. Thresholds tune throughput. Hard rules protect you. And treat the queue itself as part of the product: reviewers should see the highest risk items first, with a one screen summary and the source text one click away.

The human sign off layer

No document review agent should be the final authority on a consequential decision. The question is not whether a human signs off, it is what they are signing off on and how long it takes. In practice you want three tiers. Confirmation, where the reviewer checks a summary and the flagged items and accepts or rejects in a minute or two. Full review, where the reviewer works the file properly but starts from extracted data and cited passages rather than a blank page. Specialist review, where the file was escalated and the agent's role was purely preparatory.

Two details make this layer work. Sign off must be explicit and attributable to a named person, never implied by silence or a timeout. And rejection must be cheap and structured: capture what the agent got wrong in a field, not a comment box nobody reads. That override log is the raw material for every later improvement.

The audit trail

If you cannot reconstruct how a conclusion was reached, you do not have a reviewable system, you have an opinion generator. For regulated, legal and diligence work the trail should be designed in at the start, not retrofitted.

A sufficient trail records, for every document: the source file and its hash, when and how it arrived, what was extracted with page level citations, which standard version it was checked against, the findings and confidence, every routing decision and the rule that caused it, who reviewed it, what they changed, and when they signed off. Model and prompt versions belong there too, because "the system changed in March" needs to be answerable precisely.

The practical test: someone should be able to pick any file from six months ago and, without asking the person who built the system, see what was concluded, on what evidence, under what rules, and by whom.

How you evaluate accuracy before you trust it

You do not get to assert that an agent is accurate. You measure it, on your documents, against answers you already believe.

Start by building a gold set: a sample of real files, ideally a few hundred, spanning the messy cases as well as the clean ones, with experienced reviewers agreeing the correct answers. Where reviewers disagree with each other, record it, because human disagreement rates set the realistic ceiling for the agent and are usually higher than anyone expects.

Then measure field by field, not in aggregate. A headline accuracy figure hides the fact that dates are easy and liability positions are not. For each field you want precision and recall separately. Missing a red flag is not the same failure as raising one that does not exist, and your thresholds should reflect which hurts more in your context.

After that, run in shadow mode. The agent processes live volume, humans review as normal, and nobody sees the agent's output until after they have decided. Compare over several weeks. Set the acceptance bar before you look at the results, and keep measuring after go live. Drift is real: formats change, counterparties change templates, and a model update can shift behaviour unannounced.

Rollout sequencing

The pattern that works is boring. Pick one document type, high volume, with a written standard and a low cost of a caught error. Write that standard down first, including the parts your senior people carry in their heads, because an agent can only apply rules that exist. That exercise alone usually improves consistency.

Then move through the stages in order. Intake and classification first, since it is verifiable and low risk. Extraction with citations next, verified by sampling. Then checking against the standard, in shadow mode. Then routing, once you trust the findings. Then, and only then, narrow the human step from full review to confirmation for clean cases, keeping every escalation path intact.

Expand by document type rather than by relaxing supervision. A second type at full oversight is a better next move than the same type at half oversight.

Where AI fails here

Agents fail on poor source material. Bad scans, handwriting, photographs at an angle, dense tables and stamped or annotated pages are genuine failure modes, and text recognition errors propagate silently into extraction. Detecting low quality input and refusing it beats heroic guessing.

Agents fail on cross document reasoning. Reading one document well is largely solved. Understanding that an amendment overrides a master, that a defined term shifted between versions, that two documents contradict each other, or that individually unremarkable facts add up to a problem, is much harder and needs explicit design plus human review.

Agents fail at judgement under ambiguity. A model can tell you a file departs from your standard. It cannot tell you whether that departure matters given the relationship, the commercial context or your risk appetite. It has no stake in the outcome and should not imitate one.

Agents can be confidently wrong. Fluent, plausible, wrong. This is why verbatim citation is the core control, and why any output a person cannot trace back to source text in seconds does not belong in a review workflow.

Agents do not carry responsibility. Professional duties, privilege, confidentiality, data residency and retention obligations stay with your organisation. Solvable, but they must be solved before the first production file.

The realistic outcome is not an unstaffed review function. It is one where the queue is sorted, the facts are on the page with citations, routine files clear in minutes, difficult ones reach the right person with the groundwork done, and every conclusion can be traced. Adopt AI Partners builds these systems inside the business and stays until the team owns them.