What "99% Accurate" Means in a Data Labeling Contract

"99% accurate" means nothing until a contract says 99% of what, judged by whom, against which definition, on how many items. The same batch of labels can honestly score 99% on one definition and 85% on another. And a 100-item spot check with zero errors can only prove accuracy above 97%; proving 99% takes at least 299 items with no errors, or 473 with one.

Every labeling vendor quotes an accuracy figure, including us: Akmon guarantees 99%+. This guide explains how to pin that number down, how to audit it yourself, and how to write it into a contract so it means something when a batch goes wrong.

What can "accuracy" mean?

At least five different things. Take one batch: 1,000 street images with 20,300 objects in them. The vendor drew 20,000 boxes. 200 of those boxes are wrong (bad class or sloppy fit), spread across 150 images. 300 objects were never boxed at all.

All five are honest. A vendor quoting 99% per label isn't lying about this batch. But your model trains on images, not on individual boxes, and the 300 missed objects teach it that those objects are background.

Two rules follow:

1. Count misses. An accuracy figure that only grades labels the vendor chose to draw rewards skipping hard cases. Make sure your definition includes objects that should have been labeled and weren't.
2. Match the unit to how errors hurt you. If one wrong box ruins an image for your purpose, measure per image. If errors average out across millions of labels, per label is fine.

How should accuracy be defined for each task?

The definition has to say what counts as correct, in a way two graders would apply the same way. These are the defaults we'd start from:

Two details trip up most contracts:

• Who adjudicates. When the vendor's grader and yours disagree, someone decides. Name that person or process in advance.
• The ceiling. If two of your own experts agree only 94% of the time on a task, no vendor can hit 99% on it. Measure your own agreement on a sample first. If it's low, fix the guidelines before you hold anyone to a number.

How many items do you need to check?

More than most teams check. A random sample can only prove accuracy down to a confidence level, and small samples prove very little. This table gives the smallest random sample that demonstrates a target accuracy at 95% confidence, depending on how many errors you find in it:

Read it this way: to show a vendor is at least 99% accurate, check 299 random items and find no errors, or check 473 and find no more than one.

And here is what common spot checks actually prove:

A vendor that shows you a perfect 100-item sample has demonstrated 97%, not 99%.

For the zero-error case there's a shortcut: the sample size is ln(0.05) ÷ ln(target accuracy), which is why the old "rule of three" says 300 clean items demonstrate roughly 99%. The other cells use the exact one-sided Clopper–Pearson bound.

Three practical notes:

• Sample randomly from the whole delivery, not the first file or the vendor's chosen examples.
• Stratify if rare classes matter. A 300-item random sample of a dataset where a class is 1% of items contains about three of them, which proves nothing about that class.
• Keep sampling after the pilot. Seed 2–5% golden items with known answers into ongoing work, unmarked, so quality is measured continuously instead of once.

How do you write accuracy into a contract?

With eight clauses. Copy this template, fill in the brackets, and attach it to your statement of work. It's free to use, with any vendor.

ACCURACY SERVICE LEVEL — [Project name]

1. Definition. A label is correct when [task-specific rule, e.g. "class matches
   the guidelines and IoU with the reference box is >= 0.7"]. Objects that
   should have been labeled and were not count as errors.

2. Unit. Accuracy is measured per [label | object | item], as
   correct units / total units in the audit sample.

3. Target. [99]% or higher on each delivery.

4. Sample. Client audits a random sample of [473] units per delivery
   (or all units if fewer). The batch meets the target if the sample
   contains no more than [1] error(s).

5. Grader and adjudication. Client's reviewer grades the sample within
   [3] business days. Disputed units go to [named person/process], whose
   decision is final.

6. Remedy. If a delivery misses the target, Vendor re-labels the full
   delivery at no charge within [5] business days, and Client re-audits.
   [Two] consecutive misses allow Client to [terminate | withhold payment].

7. Guidelines. Accuracy is measured against guidelines version [x].
   Changes take effect for deliveries started after Vendor confirms receipt.
   Re-labeling past work under new guidelines is billed at [rate].

8. Reporting. Vendor delivers a QA report with every batch: units
   delivered, internal QA sample size, errors found and fixed, and
   per-class accuracy for classes listed in the guidelines.

Hold firm on clause 6, the full-delivery redo. Without it, a failed audit becomes a negotiation. Expect a fair vendor to hold firm on clause 7 in return: when you change the guidelines, re-labeling old work is your cost, not theirs.

How does Akmon measure its 99%?

We hold ourselves to the template above. Concretely:

• Before delivery. A dedicated team trained on your documentation labels to your schema, and every label passes multiple QA checks before it lands in your bucket.
• With delivery. Labels arrive in your format, in your bucket, with the QA report.
• If we miss. Under 99% accurate, we redo it at no cost.

Case study: from one pilot to four production programs

A US computer vision company came to us because its previous vendor could not hold accuracy at production standard. The work spanned road and drivable-area annotation, object detection, concrete segmentation, aerial imagery, and blueprint interpretation.

"The data is consistently accurate, and the team is extremely professional. We're impressed by how seriously the team approaches accuracy." — Co-founder, US computer vision company

The fastest way to test any of this is a pilot batch against the template above, graded by your team. Book a call and we'll start one this week.

FAQ

What does 99% accuracy mean in data labeling? Only what the contract defines: the unit (label, object, or item), what counts as correct, whether missed labels count, and how the sample is drawn. The same batch can score 99% per label and 85% per image.

How many labels should I audit to verify 99% accuracy? At 95% confidence, 299 random items with zero errors, or 473 with at most one. A clean 100-item sample only demonstrates 97%.

What IoU threshold should I use for bounding boxes? 0.5 is a common default for coarse detection; 0.7 or higher when tight boxes matter to your model. Pick one, write it down, and grade every delivery against it.

What should happen when a labeling batch fails the accuracy target? The vendor re-labels the full delivery at no charge within an agreed time, and you re-audit. Write it into the contract before the first batch.

Why can't my vendor hit 99% on our task? Check your own ceiling first. If two of your experts agree less than 99% of the time on the same items, the guidelines are ambiguous, and no vendor can exceed that agreement.

Sources

• Sample sizes: computed with the exact one-sided Clopper–Pearson bound at 95% confidence. Zero-error case: n = ln(0.05) / ln(target accuracy).
• Example batch figures (20,300 objects; 200 wrong; 300 missed) are illustrative.