Entailment Labs

Evaluation data for coding models

We write and review the tasks that teams use to test and train coding agents. Our depth is Go and Rust: defects whose cause and fix sit in different files, in containers that give the same result on every run.

Section 1: evaluation data

A second line of work, run by the same team. It is for AI labs and data companies. BPO partners can skip this page.

What we supply

Benchmark tasks

A defect, a reference solution, and tests that verify the fix without giving it away. Delivered in your task format.

Rubric reviews

Scored verdicts on code changes. Every score cites the line, log or run it rests on.

Adversarial tests

Tests written to break a candidate solution or a grader, not to confirm it.

Rubric QA

Where a rubric accepts a wrong answer or fails a right one, and the correction.

Transcript review

Coding-agent transcripts reviewed step by step, with each failure placed at the step where it happened.

Training data

Preference pairs, ranked trajectories and worked solutions, written to your specification.

Section 2: what we supply

We work in the program’s own format and environment. We do not ask anyone to adopt ours.

How we check our own work

Before we trust a test, we run it against the defect it is meant to catch. A test that passes while the defect is still there is not checking anything, so it is rewritten or removed.

Then three reviews, in order.

  1. An automated pass: build, lint, a repeat run to confirm the result does not change, and the tests run against both the broken and the fixed code.
  2. A scored review on a fixed sheet, one row per criterion, with evidence attached to each verdict.
  3. A senior sign-off on anything borderline, with the reason written down.

Nothing that describes a live task is published, on this site or anywhere else, until the program it belongs to says it may be.

Section 3: quality

We publish no acceptance rates yet. They will come from the ledger we keep from the first program, and we will not quote one before it exists.

Where we stand

ItemTodayNext
TeamSix people. Every task passes through one of us before it leaves.Contributors added through an assessment and a calibration period, sized to a program
Public workBenchmark tasks in preparationA first accepted task
SecurityTwelve written policies and a SIG/CAIQ answer bankSOC 2, not started
CompanyRegistration pendingIncorporation, then GST
PricingPer accepted task, or per reviewer-hour for review-only workFigures set from a first program's calibration batch

Section 4: status

Written down so that nobody has to discover it in a call. The table changes as each item lands.

Who this is for

Program owners at AI data companies, and research teams at labs, who need Go or Rust tasks, reviews or adversarial tests. If a program needs more people than we can staff well, we will say so on the first call.

Write to hello@entailmentlabs.com with the task format, the volume and the deadline.

Section 5: contact

We reply within two business days.