Benchmarks
Four suites that measure what a BPO actually buys: how much volume survives without a person, whether a contact was resolved or only ended, what a wrong answer costs in reviewer minutes, and whether the system is still trustworthy two months in.
The definitions are the point. Vendors quote containment and accuracy without saying what they counted, so two numbers that look comparable are not. Every metric here is defined tightly enough to put in a contract, and the data and code are public so anyone can check the definition was followed.
Section 1 — benchmarks
Data CC BY 4.0, code MIT, in nagu-io/benchmarks.
No system has been scored yet. The tables exist and are empty, and the reason each row is empty is stated on the row.
The four suites
| Suite | What it measures | Unit scored | Headline metric | Status |
|---|---|---|---|---|
| Messy Scan | What degradation does to extraction, and how much of the volume survives without a person | One document | Document-level straight-through-processing rate | not run — no model API keys, no reachable model interface |
| Honest Containment | Whether a contact was resolved, or only ended | One contact | Containment under section 3.9 | not run — no model API keys, no reachable model interface |
| Exception Economics | What automation costs when it is wrong, in reviewer minutes and money | One work item | Net cost per item at a stated confidence threshold | not run — no model API keys, no reachable model interface |
| Day-60 | Whether the system is still trustworthy two months after go-live | One deployment | Day-60 score, 0 to 100 | not run — requires a live deployment and an agreed exercise window |
Section 2 — the four suites
The status column is read from the benchmark charter. Two suites wait on model API keys; Day-60 waits on a live deployment and an agreed exercise window.
The rules this runs under
No figure is typed by hand
Every number on the benchmark site is read at build time from a results file. If a file is missing, the page says so rather than filling the gap.
A row that has not been run says so
An unrun system reads not run, with the reason. It is never an estimate, a projection, or a figure carried over from a similar system.
Every dataset rebuilds from a seed
The repository holds the generator and the SHA-256 of its output, not the output. Anyone can rebuild the set and check the hash matches.
The data is synthetic
Every document, contact and work item is generated. No partner data appears in a public set, in any form, at any time.
We score ourselves on the same page
Our own system sits in the same table as everything else, under the same definitions, with the same denominators.
Anyone may dispute a figure
The dispute process is public and its outcomes are published whichever way they go, including against us.
Section 3 — the rules
The full charter, including the five competing definitions of containment and what these benchmarks cannot tell a buyer, is on the benchmark site under Methodology.
What a partner can do with them
The harness runs against your own folder of documents, not only against ours. A pilot scope can name a suite and a threshold, so the acceptance test for the work is a number both sides agreed the meaning of before the work started. That is the reason these exist.
The bake-off kit is for the case where you are choosing between us and someone else. It scores every system through the same scorer, on the same set, and prints the table.
Section 4 — what a partner can do
Questions about a definition, or a figure you think is wrong: hello@entailmentlabs.com.
Working notes on how we measure are under Research.