Ask any administrative team where the time goes and the answer is almost always the same: documents. Open the PDF, find the number, copy it into the ERP, check it matches the order, file it, repeat. It is simple work that never ends, which is exactly why it is one of the first places where AI pays off.
What the research shows
A study published in Nature Communications in 2024 tested automatic data extraction from scientific papers, a dense type of document full of tables, units and varied phrasing (Polak and Morgan, 2024). More than the final accuracy, the interesting result was the difference made by one extra step: making the model check its own extractions with follow-up questions.
Without the verification step, precision (the share of extracted values that were correct) was 42.7%. With it, precision rose to 90.8%, while the system still captured 87.7% of the data it should have found.
A second study, which tested GPT-4 on systematic literature review tasks, reached a complementary conclusion. On data extraction, performance was moderate, and the authors advise “substantial caution” before doing away with human review (Khraisha et al., 2024).
Both studies use scientific documents, not invoices. But the lesson carries over: extraction reliability also depends, to a large extent, on how the process around the model is designed.
Why documents are a good first step
Volume is high and steady: every week brings invoices, delivery notes, proofs of delivery, contracts and requests. The rules are clear, because you know which fields to pull out, where to put them and what to compare them with. And the result can be checked: either the number is right or it isn’t, and there is almost always a second system to compare it against.
A four-step process
- Classify what comes in: invoice, credit note, delivery note, contract, complaint.
- Extract the relevant fields, recording for each value the place in the document it was read from.
- Verify, as in the study: check each extraction against the document itself and cross-check it with what already exists, the order in the ERP, the trip in the TMS, the policy in the claims system.
- Handle the exceptions: open the complaint, request the missing document, or pass the case to a person when something doesn’t add up.
Where it usually goes wrong
- Automating extraction and forgetting verification, when getting the numbers out of the PDF is half the job, and the less reliable half.
- Demanding 100% before you start, when the goal is for the person to see only the cases that need them.
- Not keeping the source of each value. Without it, checking takes as long as doing it by hand.
What changes for the team
The team’s work becomes review. Instead of opening every document, they see a list of exceptions: the invoice with a price that differs from the order, the delivery note with quantities short, the contract with a non-standard clause. It is more interesting work, and the volume can be cleared by the end of the day.
Sources
- Polak, M. P., Morgan, D. (2024). Extracting accurate materials data from research papers with conversational language models and prompt engineering. Nature Communications, 15, 1569.
- Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., Hadfield, K. (2024). Can large language models replace humans in the systematic review process?. Research Synthesis Methods.