Job Sheet Parsing: Turning Paper and PDFs into ERP Data
TL;DR
Plain OCR is a commodity: £1.13 per 1,000 pages on Azure, AWS and Google alike (July 2026). The API bill is never the problem. The problem is that on real handwritten forms the best system on the planet gets about 78% of fields right and hallucinates 6–8% of them — while a human double-keying the same sheet errs on 0.14%. Not one of Microsoft, AWS or Google publishes an accuracy figure for their OCR. Build for an exception queue, not for straight-through processing, and expect to route roughly 20% of fields to a human to hit 99% on the rest.
The job sheets come back in five states.
Some are clean PDFs, generated by a scheduling system and emailed in. Some are scans off the office multifunction device. Some are the pink carbon copy, folded into quarters, with a coffee ring. Some are photos taken on a phone in a van cab at 4:40pm with the sun coming through the windscreen. And some arrive as a WhatsApp image, compressed to death, sent to a supervisor's personal number.
Then somebody in the office types all of them into the ERP. Labour hours, parts used, a date, a signature, whether the follow-up box was ticked. Eight minutes a sheet if the writing is good.
This is the least glamorous automation problem in the mid-market and one of the most valuable. It is also the one where vendor claims and reality diverge the most, because every published accuracy number you will see was measured on clean documents. This piece is about what actually happens when you point document AI at the pink carbon copy.
£1.13
Plain OCR, per 1,000 pages
Azure, AWS and Google, July 2026
0.14%
Human double-keying error rate
Int J Med Inform meta-analysis, 2025
78%
Best AI field accuracy, real handwritten forms
arXiv 2604.16504, April 2026
0
Cloud vendors publishing an accuracy figure
Azure, AWS, Google — none
What "accuracy" means before anyone quotes you a number
Character Error Rate (CER) is what research papers report: the proportion of characters transcribed wrongly. Field accuracy is what your ERP cares about — did the part number land correctly in the part number field. There is no published formula converting one to the other, and anyone offering you a clean conversion has invented it.
Character accuracy also flatters, and the distribution matters more than the mean. The ISRI Annual Test of OCR Accuracy — still the most rigorous OCR study ever run, 2,229 pages with ground truth keyed four times by independent typists — found that "for every 1% drop in character accuracy, there is roughly a 2% decline in non-stopword accuracy", and repeatedly that "about 70% of the total errors are made on the worst 20% of the sample" (ISRI TR-93-01, TR-96-01). Stopwords are easy. Part numbers are not. And average accuracy tells you nothing about the sheets that will cost you money.
Accuracy expectations by document type
This is the table nobody publishes. The left columns are cited benchmark evidence. The right column is my own planning judgement from delivery work — not a cited figure — because no published benchmark measures "handwritten job sheet photographed in a van".
What to expect from document extraction, by what actually lands in the inbox
| Document type | Published benchmark evidence | My planning assumption (judgement, not cited) |
|---|---|---|
| Digital-native PDF generated by a system, never printed |
olmOCR-Bench: 77–86 on born-digital arXiv PDFs vs 29–50 on scans, same systems. Kleister found direct PDF text extraction beat Azure OCR on born-digital documents (81.77 vs 80.57 F1) — the authors advise not to OCR these at all. | Straight through. Parse the text layer, never the pixels. Rules and validation do the work; no model needed for most fields. |
| Scanned typed sheet flatbed, 300 dpi, printed form |
ISRI 1996: 99.49% character accuracy on clean typed legal documents, 98.83% on business letters, 95.21% on German business letters — best-in-class, same test. RealDocBench 2026: 89–96% per-field on real regulated documents. | Straight through above threshold. Expect a handful of character errors on every page even at 99.5%. Validate against master data, not against confidence alone. |
| Photo of a typed sheet phone camera, flat, decent light |
MDPBench (Mar 2026) measured the same models on digital vs photographed documents: Gemini 3 Pro dropped 5.3 points, Claude Sonnet 4.6 dropped 15.7, ChatGPT-5.2 dropped 22.6, DeepSeek-OCR dropped 38.5. Paper’s own summary: an average 17.8% drop on photographed documents. | Review the flagged fields. Model choice matters far more here than on scans — the spread between best and worst widens from 10 points to 43. |
| Handwritten entries on a printed form the standard job sheet |
CHURRO (EMNLP 2025) on one protocol: GPT-5 scored 60.7 on printed, 30.4 on handwritten. Azure OCR 71.9 → 47.7. On 49 real handwritten clinical forms (arXiv 2604.16504, Apr 2026) the best model reached 78% field accuracy once blank fields were excluded, with 6–8% of fields hallucinated. | Human confirms every handwritten field. Treat extraction as a first draft that saves typing, not as a posting decision. |
| Mixed / annotated ticks, crossings-out, margin notes |
CheckboxQA (Snowflake, Apr 2025): humans 97.5, GPT-4o 66.7, Gemini 2.0 Flash 54.4. RealDocBench key-value pair detection F1: Azure Document Intelligence 0.000, PaddleOCR-VL 0.000, dots.ocr 0.000, AWS Textract 0.121. | Do not automate. A crossed-out figure with the correction written above it is the single most reliable way to get a wrong number posted with high confidence. |
| Photo in a van cab glare, crease, angle, thumb in frame |
DocTr (arXiv 2110.12942) on 130 mobile-camera photos of warped documents: raw CER 0.68, falling to 0.20 after geometric unwarping and illumination correction. Geometry and lighting, not the camera, cause the damage. | Fix at capture, not in the pipeline. A capture app that rejects a bad photo on the spot is worth more than any model upgrade downstream. |
The single cleanest demonstration of the handwriting cliff comes from CHURRO, which ran printed and handwritten historical documents through the same systems under the same protocol. Scores are normalised Levenshtein similarity, higher is better.
Printed vs handwritten, same systems, same protocol (normalised similarity, higher = better)
GPT-5 halves. Mistral OCR more than halves. Same document class, same run — the only variable is whether a human wrote on it.
Nobody publishes an accuracy number. Nobody.
I checked the product pages and technical documentation for all three hyperscalers plus Tesseract. Azure AI Document Intelligence, AWS Textract, Google Document AI and Tesseract publish no character- or word-accuracy figure for printed text on any document set — only confidence scores, image requirements and speed benchmarks. If someone quotes you "Azure is 99.8% accurate", ask for the source; the trail usually ends at AIMultiple, whose metric is Sentence-BERT cosine similarity, not character accuracy. The one vendor that does publish is Mistral: 94.89% overall and 98.96% on "scanned" documents, on an internal, non-public test set (Mistral, 6 March 2025). Independent olmOCR-Bench scores that same product 29.3 on old scans.
The comparison that matters: machines versus the person currently doing it
Re-keying is slow and expensive. It is also, done properly, extremely accurate — the part nobody selling automation says out loud.
A meta-analysis of 93 studies pooled the error rates (International Journal of Medical Informatics, March 2025). Double data entry: 0.14% of fields. Single keying: 0.29%. Optical scanning of the OMR/ICR era: 0.74%. But human abstraction — reading a messy source document and transcribing it — pools at 6.57%, roughly 23× worse than keying.
Error rate per field, lower is better (%)
That chart says two things at once. A person carefully typing from a legible sheet beats every AI system available today by an order of magnitude. And a person squinting at bad handwriting under time pressure is in the same accuracy band as good document AI.
So the business case is not "AI is more accurate than people". It is: extraction eliminates the typing, the human keeps the judgement, and you stop paying for the typing — the same pattern that makes voice-to-CRM work in field sales.
One more finding for your design: across 104,720 fields, double-keying by a different operator caught 88.3% of errors; by the same operator, only 69.0% (Controlled Clinical Trials, October 2003). If your reviewer is the person who took the photo, you have built the weaker system.
What it actually costs
The published, pay-as-you-go rates. Sterling leads, dollars in brackets, converted at $1.3320 per £1 — the Bank of England daily spot rate for 23 July 2026 (series XUDLUSS). Rates are indicative; vendors bill in USD and the rate moves.
Document extraction pricing, pay-as-you-go, checked 25 July 2026
| Service | Per 1,000 pages | USD list | Volume break |
|---|---|---|---|
| Plain OCR — fully commoditised | |||
| Azure AI Document Intelligence — Read | £1.13 | $1.50 | £0.45 above 1M/mo |
| AWS Textract — DetectDocumentText | £1.13 | $1.50 | £0.45 above 1M/mo |
| Google Enterprise Document OCR | £1.13 | $1.50 | £0.45 above 5M/mo |
| Mistral OCR | £3.00 | $4.00 | — |
| Prebuilt invoice / receipt / expense models | |||
| Mistral Document AI | £3.75 | $5.00 | — |
| Azure DI — prebuilt models | £7.51 | $10.00 | flat |
| Google Invoice / Expense Parser | £7.51 | $10.00 | flat |
| AWS Textract — AnalyzeExpense | £7.51 | $10.00 | £6.01 above 1M/mo |
| Custom / generic form extraction — where job sheets land | |||
| Azure DI — custom extraction | £22.52 | $30.00 | flat |
| Google Custom Extractor / Form Parser | £22.52 | $30.00 | £15.02 above 1M/mo |
| AWS Textract — AnalyzeDocument Forms | £37.54 | $50.00 | £30.03 above 1M/mo |
| AWS Forms + Custom Queries + Tables | £60.06 | $80.00 | £45.05 above 1M/mo |
| Add-ons and fixed costs that surprise people | |||
| AWS — Signatures detection | £2.63 | $3.50 | £1.05 above 1M/mo |
| Google add-ons (checkbox extraction, maths OCR, font style) | £4.50 | $6.00 | flat |
| Azure add-ons (high-resolution, formula, font property) | £4.50 | $6.00 | flat |
| Azure — custom neural model training | £2.25 / hour | $3.00/hr | first 10 hrs/mo free |
| Google — deployed processor hosting | £329 / year each | $438/yr | per deployed version |
Now the arithmetic that decides the project.
Take an operation putting through 500 job sheets a week, 48 weeks a year — 24,000 pages. Run every one through custom extraction at £22.52 per 1,000: £541 a year. Add signature detection and checkbox extraction and you are still under £750. The same 24,000 sheets re-keyed at eight minutes each is 3,200 hours, or £70,400 a year at £22 an hour fully loaded.
(Illustrative. Assumptions: 500 sheets/week, 48 weeks, 8 minutes per sheet, £22/hour blended cost including employer NICs. Not a client result — substitute your own numbers.)
The API cost is 0.8% of the labour it displaces. It is noise. Which means the cost of a job sheet parsing project is entirely the integration, the review workflow and the exception handling — not the model. Every hour spent choosing between Azure and Google is an hour not spent on the thing that determines whether it works.
Where LLMs help, and where they should be nowhere near your data
The useful framing is not "LLMs versus OCR". It is: which failure mode can you tolerate.
Classical OCR and layout models fail loudly. They return garbage, low confidence, or nothing. You can see it. Vision language models fail quietly — a plausible, well-formatted, confidently-wrong value, because the same language prior that helps them read messy prose actively fabricates on unpredictable strings.
The evidence is unambiguous. CHURRO's error analysis found 36% of a small-model sample contained major hallucinations, arising "when a model cannot reliably recognize the text and instead generates content that seems contextually plausible" (arXiv 2509.19768) — unable to transcribe an 18th-century letter, the model produced "Dit is een brief", "This is a letter". On real handwritten clinical forms, hallucination rates ran 6% to 17% of fields, and models "do not flag uncertainty. They will either return a blank value, infer the closest plausible alternative, or hallucinate some output entirely" (arXiv 2604.16504). METATR (TEKLIA, May 2026) recorded CER above 100% for several open models — arithmetic proof they emitted more text than the document contained.
So:
Use a VLM for photographed and low-quality documents, where the vendor-run OmniAI benchmark found "VLMs are also more predictable on photos and low quality scans"; for unusual layouts; for handwriting where a specialist model is not viable; and — the highest-value use — for classification and routing.
Use classical OCR and layout models for high-density typed pages, where OmniAI found traditional models outperform; for anything you can validate deterministically afterwards; and wherever you need a confidence score you can act on. Note that the specialist HTR product Transkribus scored 9.13 CER on modern IAM handwriting — five times worse than a general LLM — but beat every LLM on degraded historical German (arXiv 2503.15195). Specialists win where they are specialised and lose everywhere else.
Use neither for digital-native PDFs. Read the text layer.
Confidence scores will not save you
Researchers tested token-level log-probabilities, verbalised confidence and multi-sample self-consistency as gates on LLM extraction, and found they "all collapse toward all-positive behaviour at practical thresholds, offering no reliable separation between trustworthy and untrustworthy extractions" (arXiv 2606.24420, June 2026). Their purpose-built verifier reached 99.1% accuracy — but only at 80% coverage, meaning one field in five was deferred to a human. Microsoft says something similar in plainer language: target "80% or higher", but "for more sensitive cases, like financial or medical records, we recommend a score of close to 100%" — and note that "custom neural and generative models don't provide accuracy scores during training" (Microsoft Learn, 8 April 2026).
The failure modes, specifically
Tick boxes. The worst-performing common element on any job sheet. CheckboxQA's named failure modes describe exactly what you will see: defaulting to textual clues instead of checkbox states (the model reads the label beside the box and ignores whether it is ticked), selecting all possible options in a list, and ignoring table structure for checkbox fields (arXiv 2504.10419).
Carbon copies. ISRI named the causes 30 years ago and nothing has changed: broken and touching characters from photocopying, shaded backgrounds, creases, speckling. These are precisely the pages in the worst 20% that generate 70% of your errors.
Part numbers. The strongest evidence that alphanumeric codes are a known-hard capture problem is a regulation. US 49 CFR § 565.13(g) specifies that every VIN character must come from the set [ABCDEFGHJKLMNPRSTUVWXYZ] — I, O and Q are deliberately absent, being the three letters confusable with 1, 0 and 0 — and adds a check digit defined as existing "to verify the accuracy of the transcription of the vehicle identification number" (eCFR). A standards body designed around this in 1980. If your part numbers mix O and 0, no model will fully solve it. Validate every extracted code against the item master and reject anything that does not match.
Units and signatures. A job sheet reading "2.5" next to a part could mean metres, each, or hours; the model will pick one. That is a form-design problem, not an AI problem, and the cheapest thing on this list to fix (practitioner observation, no citation). And AWS's £2.63 per 1,000 pages for signature detection buys detection that a mark is present, not verification of whose it is — enough for sign-off, not enough for anything contractual.
UK versus US dates. The one that quietly corrupts data for months. Azure's prebuilt-invoice model types InvoiceDate and DueDate as a date rather than a string, and ships en-US, en-GB, en-AU, en-CA and en-IN locales in the same model. Microsoft's documented example value is 11/15/2019 — unambiguous only because 15 is greater than 12. AWS goes the other way: AnalyzeExpense returns INVOICE_RECEIPT_DATE as detected text, with currency normalisation but no date normalisation and no locale parameter (AWS documentation). Every date between the 1st and the 12th is ambiguous, will parse without error, and will be wrong half the time if the locale is wrong. Set it explicitly, then test with a sheet dated the 3rd.
How the data actually lands in the ERP
Three ERPs dominate the UK mid-market and they behave very differently. This is the part of the project that consumes the budget.
Capture to posted ERP transaction, with the gate in the right place
Business Central is the easiest of the three to do properly, because Microsoft has already decided the pattern for you: stage in a journal, validate, post as a separate action, and wrap the batch in Isolation: snapshot for all-or-nothing. SAP Business One is the most flexible and the most dangerous — no published rate limits, no transactions spanning requests, and fail-fast batching that leaves partial state. Sage 200 is the hardest, and not for the reason people expect: the rate limit is generous, but there is no unattended authentication path at all.
The vendor landscape, verified
Three names come up repeatedly in buyer research on SAP and ERP document automation: Conduct, Nova Intelligence and Transformance AI. All three are real companies. None is a UK mid-market job sheet capture vendor, and two do not touch documents at all. Saying so is more useful than pretending otherwise.
Conduct is Conduct AI Ltd, Companies House 15396489, incorporated 8 January 2024, London, around 30 people, $60m Series A from Index Ventures, ICONIQ and SAP. Its own words: "Conduct lets IT teams understand, modernise, and run their enterprise systems with AI — fast and with confidence. Starting with SAP." It maps SAP customisations and grades clean core for S/4HANA migrations. No document extraction, OCR or capture capability appears anywhere on its site.
Nova Intelligence is Nova AI Software Inc, San Francisco, around 20 people, $31.5m Series A in May 2026: "the agentic AI platform for SAP custom code modernization and S/4HANA transformation." A direct competitor to Conduct, not to a capture vendor. No document extraction capability. Caution: a UK-registered NOVA INTELLIGENCE LIMITED (CH 16280548, Brighton) exists but is an unrelated management consultancy under proposal to strike off.
Transformance AI is Transformance Solutions GmbH, Hamburg, HRB 194363 — the only one of the three that genuinely does document extraction, but of remittance advices in an order-to-cash context: "The autonomous order-to-cash agent for finance teams." Its DocSense component claims "99.7% accuracy on structured remittance data" at 2,000 pages a minute, using a vision language model rather than classical OCR. A vendor claim on structured documents — precisely the clean-document caveat this article is about. Supported ERPs are SAP, Oracle, NetSuite and Dynamics: not SAP Business One, not Sage 200, not Business Central.
If you are actually shopping, the relevant vendors are different ones. Continia Document Capture for Business Central publishes a full GBP tier table and is the only vendor I found listing delivery notes as a supported document type. Equisys Zetadocs Capture (CH 02120744, London, trading since 1987) publishes £105/month for five users plus £0.20 per document. Sicon Document Automation is Sage 200 only, with the clearest per-document pricing in the category — £0.50 at 1,000 documents, falling to £0.29 at 200,000, prepaid, no expiry. Zahara covers Sage 50, Sage 200, Business Central and Xero at £159 to £499 per month ex VAT.
The negative finding that matters most
No UK field service management platform does OCR of completed paper job sheets. I checked Joblogic, BigChange, Commusoft, Klipboard and simPRO. Every one replaces the paper form with a mobile app form filled in on site. Where OCR does exist — Joblogic's expense scanner, BigChange's supplier invoice reader — it is pointed at accounts payable documents, never at the field paperwork. Klipboard's "digitise your paper-based forms" means rebuilding a blank form as a digital template, not reading a completed one. (Also worth knowing: BigChange and simPRO are now the same company, both owned by Simpro Group.) If your engineers still fill in paper, no FSM vendor will read it for you. You either change the capture method or build the parsing yourself.
What good looks like
Set the threshold per field, not per document. A job number that fails a lookup against open jobs is worthless at any confidence. A free-text comment at 60% is fine. One global threshold is the most common design mistake I see.
Validate deterministically wherever you can. Part numbers against the item master. Job numbers against open jobs. Engineer names against the employee list. Hours against the shift pattern. Dates against the job's scheduled window. Every deterministic check removes a class of error no confidence score would have caught — your only real defence against a confidently hallucinated value.
Build the exception queue first. Not the model. The queue is the product. It shows the original image beside the extracted fields, lets a human correct in seconds without leaving the screen, and records what was corrected. If reviewing an exception takes longer than re-keying the sheet, you have built something worse than what you replaced.
Plan for roughly 20% of fields to reach a human, at least in year one. That is what the coverage-versus-accuracy research implies, and a far more honest starting assumption than any vendor's straight-through pitch. The reviewer should not be whoever captured the sheet.
Measure four things weekly. Straight-through rate. Correction rate on the fields that did go straight through — sample them, or you are flying blind. Median seconds per exception. And escaped errors found downstream, the only metric that tells you whether your thresholds are right.
Stage, then post. Draft, journal or your own staging table — never straight to the ledger. It is the difference between a bad batch being a five-minute cleanup and a month-end problem.
If that sounds like more process design than model selection, that is the point, and the same reason most AI projects fail on process rather than technology.
Methodology and sources
Pricing. Read on 25 July 2026 from the vendors' own pages: aws.amazon.com/textract/pricing, cloud.google.com/document-ai/pricing, mistral.ai/pricing/api, and — for Azure — Microsoft's own Azure Retail Prices API at prices.azure.com. The Azure public pricing page renders figures via JavaScript and returns placeholders to a non-browser client, so the API was used instead; it is first-party but a different surface from the marketing page, so open the pricing page in a browser if you need a screenshot for a board pack. AWS figures were cross-checked against the AWS Price List API and agree with the HTML page. None of the three displays a "last updated" date on its pricing page.
FX. All conversions use $1.3320 per £1, the Bank of England daily spot rate (series XUDLUSS) for 23 July 2026, the most recent published at the time of writing. The Bank's disclaimer applies: these "are not official rates and are no more authoritative than that of any commercial bank operating in the London foreign exchange market." Vendors bill in USD. Treat every sterling figure as indicative.
Cited benchmark versus practitioner judgement. Everything in the "published benchmark evidence" column of Exhibit 1, every figure in Exhibit 2, and every ERP limit in Exhibit 3 is sourced and linked. The "planning assumption" column of Exhibit 1, the units failure mode, and the recommendations in "What good looks like" are my own judgement from delivery work and carry no citation. The worked example is labelled illustrative with assumptions stated; it is not a client outcome.
Vendor verification. Conduct, Nova Intelligence and Transformance AI were each checked against their own websites plus Companies House or the German commercial register. Where their capabilities differ from what buyer research implies, I have said so.
What I could not verify. Google publishes no pricing for Document AI Workbench, uptraining or human-in-the-loop, and no Document AI free tier. Sage's developer.sage.com is behind Cloudflare and returned 403 to every method attempted, so all Sage figures come from Sage's own UK knowledge base and desktop help. SAP publishes no rate limits whatsoever for Service Layer — a documentation gap, not a finding of generosity. Middleware connector coverage for Codeless Platforms, Boomi, Workato and Celigo could not be confirmed from primary sources.
What I excluded. A circulating claim that "GPT-5 leads the IAM handwriting benchmark at 1.22% CER" does not appear on the leaderboard it cites and appears fabricated. "OCR is 99% accurate" has no traceable primary source. "1% manual data entry error rate" is not supported as stated — the pooled figures are 0.29% for keying and 6.57% for abstraction, and the folklore number matches neither.
What to do on Monday
Sort one week's job sheets into the five buckets in Exhibit 1. Not a sample — a full week. The mix is the single input determining whether this is a two-month build or a six-month one, and almost nobody knows their own mix. If more than a third are handwritten or annotated, plan a review workflow from day one rather than hoping the model improves.
Time what the office does with them today. Ten sheets, end to end. Count the fields. Find out what happens when the writing is illegible — whatever that process is, it is already your exception queue. A classic ghost workflow: real work, real hours, invisible to every system.
Change the capture before you buy anything. A form redesign that prints the units on the sheet, boxes the part numbers, and replaces tick boxes with a single circled option will do more for accuracy than any model. Same for a capture app that rejects a blurred photo on the spot. Both are cheaper than the integration and both make the integration work better.
Then price the integration, not the extraction. £541 a year of API calls is not a business case decision. Three weeks of Sage 200 authentication work is. For the wider picture on where this sits against your other options, the hidden waste audit maps admin burden across the operation in about five minutes.
Frequently Asked Questions
How accurate is OCR on handwritten job sheets?
Considerably worse than on printed ones, and worse than any vendor implies. On one controlled comparison GPT-5 scored 60.7 on printed documents and 30.4 on handwritten; Azure OCR fell from 71.9 to 47.7 (CHURRO, EMNLP 2025). On 49 real handwritten clinical forms the best of 17 models reached 78% field accuracy excluding blanks, and hallucinated 6–8% of fields (arXiv 2604.16504). Plan for a human to confirm every handwritten field.
What does document extraction cost per 1,000 pages?
Plain OCR is £1.13 ($1.50) across Azure, AWS and Google alike, falling to £0.45 at high volume. Prebuilt invoice models are £7.51 ($10). Custom form extraction — where job sheets sit — is £22.52 ($30) on Azure and Google, £37.54 ($50) on AWS Textract Forms (July 2026, at $1.3320/£1). At 24,000 sheets a year that is roughly £541. The API is never the expensive part.
Should we use an LLM or traditional OCR for job sheets?
Both, for different things. Vision language models handle photographed and low-quality documents better and are the right tool for classifying which sheet is which — but they fail silently, returning plausible wrong values. Classical OCR and layout models fail loudly, give confidence scores you can act on, and outperform on high-density typed pages. For digital-native PDFs use neither: read the text layer directly.
Can extracted data post straight into Sage 200 or Business Central?
Technically yes, practically no. Business Central mandates staging — Microsoft states you "can't import data to tables that contain posted entries" and must use journals. SAP Business One offers Drafts plus a documented preview action. Sage 200's harder problem is authentication: Sage states it "requires user interaction… This cannot be circumnavigated, or automated", the binding constraint on any lights-out pipeline.
Why do tick boxes cause so many errors?
Because models read the label rather than the box. CheckboxQA measured humans at 97.5 against GPT-4o at 66.7 and Gemini 2.0 Flash at 54.4, naming the failure modes as defaulting to textual clues instead of checkbox states and selecting all options in a list (arXiv 2504.10419). Key-value pair detection F1 on real documents was 0.000 for both Azure Document Intelligence and PaddleOCR-VL.
Is AI more accurate than the person currently typing these in?
No — not from a legible sheet. Pooled across 93 studies, double data entry errs on 0.14% of fields and single keying on 0.29% (Int J Med Inform, March 2025). The best document AI on real regulated documents is around 4%. The honest case for automation is speed and cost, not accuracy — with one exception: human abstraction from messy source documents pools at 6.57%, which is where good AI becomes competitive.
What confidence threshold should we set?
Not one — set them per field, and do not trust them alone. Microsoft recommends 80% or higher generally and "close to 100%" for financial records, and notes custom neural and generative models give no accuracy score during training. Research published in June 2026 found log-probabilities, verbalised confidence and self-consistency "all collapse toward all-positive behaviour at practical thresholds" (arXiv 2606.24420); the same work hit 99.1% accuracy only by deferring 20% of fields to humans. Deterministic validation against your item master catches what confidence never will.
Related Reading
- AI for Distribution Companies: A Practical Guide for UK Wholesalers
- Voice-to-CRM: The Complete Guide for UK Field Sales Teams
- Ghost Workflows: The Manual Tasks Nobody Counted
- Where Margin Dies: The Sales-to-Ops Handoff
- The Hidden Cost of Manual Processes in UK Manufacturing
- The Real Cost of Bad CRM Data in UK Field Sales
- AI Quoting for Manufacturers: A Practical Guide
- Why 70% of AI Projects Fail — And What the 30% Do Differently
Want to know whether job sheet parsing is your highest-value first project? The Hidden Waste Audit maps where the admin hours actually sit across your operation. Five minutes, no pitch. Or book a 30-minute call and bring a week's worth of sheets.