The Review Queue Is the Product: How Many Humans Your AI Agent Needs

An AI agent that sends 10,000 decisions a day, with 12% routed to a person for approval, needs about 14 full-time reviewers to run around the clock, not the 4 that back-of-envelope math suggests. The gap comes from queueing, traffic peaks, breaks, and the simple fact that a 24/7 seat takes several people to fill.

Most guides to human-in-the-loop AI stop at choosing a confidence threshold. Teams that ship learn the threshold is the easy part. The queue behind it is what breaks, usually on the first busy Monday. This guide sizes that queue from first principles and includes a calculator for your own numbers.

Why does a single confidence threshold break?

Because it collapses three different decisions into one. A production agent needs two thresholds and three outcomes:

Risk matters as much as confidence. A common production pattern lets agents act alone on low-risk, reversible actions, sends moderate-risk actions to an approval queue, and requires explicit sign-off on anything irreversible, regardless of how sure the model claims to be. Refunds, account changes, and anything a regulator might ask about belong in the last group. Since August 2, 2026, the EU AI Act's obligations for high-risk systems make that documented human oversight a legal question, not just a product one.

One more design rule: the approval gate belongs in your application code, not in the model's prompt. A model can be talked out of an instruction to ask permission. It cannot be talked past a function that refuses to execute without an approval token.

Where you set the middle band decides your escalation rate. Your escalation rate decides your queue. And the queue is bounded by something most teams never model: how many items a staffed team can clear per hour. A common planning target is a 10–15% escalation rate. If your first pass sends 30% to review, the threshold isn't the problem; the model isn't ready for that deployment yet.

How many reviewers do you need? Start from the queue

Take an agent handling 10,000 decisions a day, with 12% escalated, and 90 seconds per review. The naive estimate is one line:

Now staff it properly. Assume traffic doubles for a four-hour peak each day, you want 90% of escalations answered within two minutes, and coverage runs 24/7:

That's 56 seat-hours of coverage a day, or 392 a week. Reviewers spend about 30% of paid time on breaks, training, meetings, and absence, so filling those seats takes 560 paid hours a week. At 40 hours each, that's 14 people, 3.7x the naive estimate.

None of these assumptions is aggressive. Two minutes is a slow answer for a customer waiting on a chat agent. Thirty percent shrinkage is a common planning assumption for staffed queues. And a doubling at peak is mild for anything consumer-facing.

What does the naive math miss?

Four things, and each one alone can double the answer.

1. Queueing. Escalations don't arrive evenly; they cluster. If two arrive in the same second and one reviewer is free, one waits. Keeping wait times short means keeping some reviewers idle some of the time. The standard model for this is Erlang C, the same math call centers have used for a century. We also cap reviewer utilization at 85%, because people who are busy every single minute get slower and sloppier.
2. Peaks. You staff for the busiest hour, not the average one. Monday mornings, end of month, the hour after a marketing email.
3. Shrinkage. Paid hours are not seat hours. Breaks, training on guideline changes, team meetings, sick days, and holidays take a large bite.
4. Coverage. One seat filled 24 hours a day, 7 days a week is 168 seat-hours. After 30% shrinkage that's 240 paid hours, or six full-time people for a single chair.

That last point is the one that surprises founders most. The first reviewer on a 24/7 queue costs you six hires. Below a certain volume, round-the-clock coverage, not workload, sets your headcount.

What actually moves headcount?

At small volumes, less than you'd think. Here is the same 10,000-a-day agent at different escalation rates, all other assumptions unchanged:

Three lessons from the table:

• Staffing is lumpy. Cutting escalation from 12% to 6% halves the review work but saves only one person, because you still need two people in seats at 3 a.m. Model improvements pay off in steps, not smoothly.
• Coverage hours are a bigger lever than accuracy at this scale. The same 12% workload, squeezed into 10 hours of coverage a day, seven days a week, needs 12 people instead of 14. If your users are in one time zone, ask whether escalations at 3 a.m. can wait until 7.
• Scale changes the picture. At 100,000 decisions a day and 12% escalation, the naive estimate is 38 people and the real answer is 90. That's 2.4x instead of 3.7x, because larger pools absorb random bursts more efficiently. At that scale, every point of escalation you remove saves real headcount.

Time per review matters too. Doubling review time from 90 to 180 seconds takes the base case from 14 people to 27. A better reviewer screen, covered below, is often the cheapest headcount reduction available.

Size your own queue

Put in your volume, escalation rate, and review time. The calculator staffs off-peak and peak hours separately with Erlang C, applies shrinkage and coverage, and shows the gap against the naive estimate.

Two inputs deserve care. Seconds per review should come from timing real reviewers on real items, including reading the evidence, not from a guess. Answer within is a product decision: how long can the person on the other end of your agent wait?

What should the reviewer see?

Everything needed to decide in one screen, and nothing else. Every extra click is seconds per review, and seconds per review are headcount. A good review item shows:

• The proposed action, exactly as it would go out: the reply, the refund amount, the label, the field values.
• The evidence the model used: the source document, the extracted fields highlighted in place, the conversation so far.
• Why it was escalated: low confidence, a high-value amount, a sensitive topic, or a category that always gets a look.
• Three buttons: approve, edit then approve, reject. Edits and rejections require a reason code from a short list.

Those reason codes are the most valuable data your queue produces. They tell you, in aggregate, exactly what the model gets wrong. That's your next fine-tuning set and your next guideline fix.

What happens when nobody answers?

Decide this before launch, per action type, and write it into code:

The worst outcome is the one teams discover by accident: the review request times out silently, and the agent ships its low-confidence answer as if nothing happened.

How do you bring escalation down without flying blind?

The goal of a review queue is usually to shrink it. Do that in steps, and keep watching what the model does on its own.

1. Start wide. At launch, review more than you think you need, sometimes every output. You're buying ground truth about how the model fails on real traffic.
2. Feed corrections back. Every edit and rejection, with its reason code, becomes training or evaluation data. Retrain or adjust prompts on a fixed cadence, not ad hoc.
3. Widen the auto-approve band one category at a time. Move a category out of review only when the reviewers' approve-as-is rate on it has been high for several weeks.
4. Audit what you no longer review. Send a random 1–2% of auto-approved outputs to reviewers anyway. Without this, you only see errors in the band you're watching, and a model regression in the confident band reaches customers unseen.
5. Re-plan staffing at each step. Because staffing is lumpy, a lower escalation rate only saves money once it crosses a seat boundary. Use the calculator to see where those boundaries are.

Some workflows never get to zero, and shouldn't. Wayfair's logistics team runs a fine-tuned model on a live queue where every output gets a decision from an Akmon reviewer before it goes out. Wayfair onboarded in under a month, and the program has since expanded. The human isn't a temporary crutch there; it's what lets a model run in a process that has to be right every time.

Should you staff the queue yourself?

Staff it in-house when the queue is small and business-hours only, when reviewers need credentials you employ (licensed clinicians, attorneys), or when the work is so tied to internal judgment that writing it down isn't possible yet.

Consider a managed team when you need 24/7 coverage, when volume swings week to week, or when the people currently reviewing are your engineers. The six-hire floor for a single round-the-clock seat is where most startups feel it first.

How Akmon runs live review

Akmon puts trained reviewers between your model and the world, in your stack and on your model:

• You set the threshold. The model routes what it isn't sure about to us; we approve, correct, or reject before anything goes out.
• Coverage that scales on short notice. 24/7 shifts, capacity for 100,000+ outputs a day, and 30 reviewers spun up over a weekend when a client needed them.
• Dashboards you didn't have before. Every reviewer, in real time, with QA and agreement stats.
• Three ways in. Run your existing review operation better, take it over from your BPO with no gap in coverage, or replace it with a model on the routine work and our people approving the rest. Wayfair did the last one in under a month.

We'll size your queue with you, using your real volume and timings, and staff it this week. Book a call.

FAQ

How many human reviewers does an AI agent need? Multiply daily decisions by the escalation rate and the seconds per review to get review hours, then staff for peaks, queueing, shrinkage, and coverage hours. For 10,000 decisions a day at 12% escalation and 90 seconds a review, that's about 14 people for 24/7 coverage.

What escalation rate should I target? A common planning target is 10–15%. Set it by working backward from the review capacity you can staff, not forward from the model's confidence scores.

What is a good confidence threshold for human review? There isn't one number. Use two thresholds, one for acting alone and one for declining, and route the middle band to review. Then adjust both until the escalation volume fits your queue.

What is Erlang C, and why use it for AI review? It's the standard formula for how many staff a queue needs to answer a given share of arrivals within a target wait. Escalations from an agent arrive like calls to a call center, randomly and in bursts, so the same math applies.

Should AI agents ever auto-approve when reviewers are slow? Only for low-value, reversible actions, and with after-the-fact review. Irreversible or high-value actions should wait for a person.

Sources

• Tian Pan: the two-threshold problem in confidence-gated AI (three outcomes; queue throughput as the constraint; 30% escalation signal)
• Galileo: human-in-the-loop oversight for AI agents (10–15% escalation target)
• Viviscape: human-AI handoff design (three-tier risk pattern; EU AI Act date)
• eesel: AI escalation design (approval gate outside the prompt)
• Gravitee: human review workflows (signed approvals; timeout behavior)
• Akmon (Wayfair program; capacity figures)
• Staffing figures: Akmon's calculator (Erlang C, 85% utilization cap, 30% shrinkage), reproducible in the embed above.