Every partner asks for an accuracy figure, and every vendor has one ready. The figures are rarely comparable, because accuracy depends on what is counted, on what sample, and against which truth. This note says how we count, so that a figure in one of our reports means the same thing in the next one.
Field accuracy, per field
The unit of measurement is a field on a document: an invoice number, a due date, a line quantity. A field is correct when the extracted value matches the ground truth after normalisation. Normalisation is written down per field: dates are compared as dates, amounts as numbers to the precision the process uses, identifiers after trimming whitespace and case. A value that is right but formatted differently is correct. A value that is missing when the document has it is wrong. A value that is present when the document has none is wrong.
We report accuracy per field, not only overall. An overall figure hides the field that matters. A system that reads the vendor name perfectly and the total amount badly is not an accurate system for accounts payable, whatever the average says.
Document accuracy
A document is correct when every field the process needs is correct. This figure is always lower than any field figure and it is the one the process owner should look at, because it says how many documents can pass without a person touching them.
Exception rate
The share of items that went to the human review queue, and why: low confidence, a failed rule, or the random sample. This is reported beside accuracy, never instead of it. A system can reach any accuracy figure by sending everything to a person; the exception rate is what stops that from being a trick.
Confidence calibration
Each extracted field carries a confidence figure. We check that the figure means what it says: among fields reported at a given confidence, roughly that share should be correct. Where it is not, the threshold for review is set from the measured relationship, not from the reported number. The calibration check is repeated every month under the operations retainer, because it is one of the first things to drift.
The sample and the truth
Accuracy is measured on the partner's own documents, chosen to cover the suppliers, layouts and periods the process sees. Ground truth is the partner's historical output, corrected where the pilot finds it wrong. The report states the sample size, how it was selected, and how many ground-truth values were corrected, so that the reader can judge the figure.
We do not measure on our own documents and we do not quote a figure before the pilot. The synthetic invoice on our home page is a demonstration of what the marks mean, not a measurement.
What the SOW fixes
The SOW names the fields, the definitions, the sample, the accuracy floor per field and the exception-rate ceiling. Every model change afterwards is gated on the evaluation set built in the pilot, against those floors. If a change lowers a field below its floor, it does not ship. If production drifts below the floor, that is a Sev 2 incident and the monthly report says so.
Why we write this down
Because a partner will be asked by their client how accurate the system is, and the honest answer has four parts: which fields, on what sample, against which truth, and how many items a person saw. A single number is not an answer. A table with its basis is.