Evaluation data for coding models
We write and review the tasks that teams use to test and train coding agents. Our depth is Go and Rust: defects whose cause and fix sit in different files, in containers that give the same result on every run.
Section 1: evaluation data
A second line of work, run by the same team. It is for AI labs and data companies. BPO partners can skip this page.
What we supply
Benchmark tasks
A defect, a reference solution, and tests that verify the fix without giving it away. Delivered in your task format.
Rubric reviews
Scored verdicts on code changes. Every score cites the line, log or run it rests on.
Adversarial tests
Tests written to break a candidate solution or a grader, not to confirm it.
Rubric QA
Where a rubric accepts a wrong answer or fails a right one, and the correction.
Transcript review
Coding-agent transcripts reviewed step by step, with each failure placed at the step where it happened.
Training data
Preference pairs, ranked trajectories and worked solutions, written to your specification.
Section 2: what we supply
We work in the program’s own format and environment. We do not ask anyone to adopt ours.
How we check our own work
Before we trust a test, we run it against the defect it is meant to catch. A test that passes while the defect is still there is not checking anything, so it is rewritten or removed.
Then three reviews, in order.
- An automated pass: build, lint, a repeat run to confirm the result does not change, and the tests run against both the broken and the fixed code.
- A scored review on a fixed sheet, one row per criterion, with evidence attached to each verdict.
- A senior sign-off on anything borderline, with the reason written down.
Nothing that describes a live task is published, on this site or anywhere else, until the program it belongs to says it may be.
Section 3: quality
We publish no acceptance rates yet. They will come from the ledger we keep from the first program, and we will not quote one before it exists.
Where we stand
| Item | Today | Next |
|---|---|---|
| Team | Six people. Every task passes through one of us before it leaves. | Contributors added through an assessment and a calibration period, sized to a program |
| Public work | Benchmark tasks in preparation | A first accepted task |
| Security | Twelve written policies and a SIG/CAIQ answer bank | SOC 2, not started |
| Company | Registration pending | Incorporation, then GST |
| Pricing | Per accepted task, or per reviewer-hour for review-only work | Figures set from a first program's calibration batch |
Section 4: status
Written down so that nobody has to discover it in a call. The table changes as each item lands.
Who this is for
Program owners at AI data companies, and research teams at labs, who need Go or Rust tasks, reviews or adversarial tests. If a program needs more people than we can staff well, we will say so on the first call.
Write to hello@entailmentlabs.com with the task format, the volume and the deadline.
Section 5: contact
We reply within two business days.