Notes from the human layer.

What we learn running annotation, live review and data capture for teams shipping models. Written for the people who have to make the call — founders, CTOs, and whoever owns model quality.

Latest

Calibration chart comparing 92.7% raw agreement with a Cohen’s kappa of 0.68 for an LLM judge against human graders

Model audit

9 min read

Your LLM Judge Needs a Human Anchor: Calibrating LLM-as-a-Judge

How to calibrate an LLM judge against human labels: build a golden set, measure Cohen's kappa, and fix a judge that fails, with tested code.

Annotated street image with a missed object, showing 99% accuracy per label but 85% per image

Annotation

7 min read

What "99% Accurate" Means in a Data Labeling Contract

How to define labeling accuracy, how many items to audit to prove it, and a free eight-clause SLA template you can use with any vendor.

Grid of AI agent decisions with 12% escalated to review, showing 4 reviewers by naive math versus 14 needed for 24/7 coverage

Live ops

9 min read

The Review Queue Is the Product: How Many Humans Your AI Agent Needs

How to size the human review queue behind an AI agent from first principles, with an Erlang C calculator for your own volume and escalation rate.

Annotation

7 min read

Cost Per Usable Label: How to Compare Data Annotation Quotes

Why the cheapest labeling quote can cost 3x more once your review time and rework are counted, with a calculator to compare real quotes.

Diagram of a retired crowd-labeling queue handing off to a managed ground-truth team, with “Mechanical Turk is gone” headline

Annotation

8 min read

Mechanical Turk Is Gone. What Your Ground Truth and A2I Workflows Need Next

How to find where Mechanical Turk still runs in your stack, pick a replacement workforce, and migrate Ground Truth and A2I in one week.

Contact

Rather just ask us directly?

Tell us what you are working on. We start within hours of kickoff.

Book a call →

Live within hours of kickoff. First production batch within a day.