Measured evidence · July 2026
What multi-agent evaluation actually costs: a measured study
We reconstructed a production-shaped, multi-document quality-gate workload on a fully synthetic corpus, then ran every competing agent configuration on the same model at temperature 0 and counted every token from the API usage records. The result is a like-for-like cost ladder for single-pass evaluation, growing-context agentic evaluation, and self-consistency voting, with and without prompt caching. These measurements power the defaults in our AI Agent ROI & TCO Calculator.
The short answer
On an identical 11-document evaluation workload, a single-pass multi-viewpoint evaluation cost $1.34 per cycle and hit output caps. A growing-context agentic configuration cost $4.64 per cycle, replaying 3.6x the input tokens of a single pass. Pushing agents toward deterministic-grade reliability with majority voting and a verification pass cost $10.63 to $15.27 per cycle, took tens of minutes, and still left roughly a 10% error rate on borderline decisions. Prompt caching compresses these numbers to $0.64, $1.85, and $4.35 to $6.20 respectively, but it cannot change the ordering, because voting multiplies output tokens and output tokens are never cached.
Method
- Workload. A synthetic corpus of 11 software design documents (about 130,000 characters) with a 58-item quality checklist, requirement traceability, and identifier contracts, generated for this study. No customer data was used.
- Ground truth. A deterministic rule implementation of the checklist was verified first: a healthy corpus variant returns pass with zero findings, and a degraded variant (one truncated document, one stubbed core document, two broken must-priority traces) returns exactly 5 findings at coverage 0.947. All agent configurations were graded against this ground truth.
- Measurement. Every configuration ran on the same commercial model at temperature 0. Token counts come from the API usage object of each call, not from estimates. Costs use list pricing: $3.00 per million input tokens, $15.00 per million output tokens; the cached scenario bills cache reads near 0.1x with a small write premium.
- Configurations. (1) Single-pass evaluation: 12 evaluation viewpoints, each reading the full corpus once. (2) Agentic evaluation: each viewpoint as a 4-turn growing-context agent. (3) Parity attempt: 2 to 3 independent agentic passes with majority voting plus a single-pass verification sweep.
Measured results
| Configuration | Tokens per cycle | Cost per cycle (uncached) | Cost per cycle (cached) | Reliability observed |
|---|---|---|---|---|
| Single-pass evaluation (12 viewpoints) | 297.9K in / 30.0K out | $1.34 | $0.64 | Outputs hit token caps; misses leave no recorded reason |
| Agentic evaluation (4-turn agents x12) | 1.074M in / 94.8K out | $4.64 | $1.85 | 3.6x input replay measured; single-run flip rate ~20% on borderline decisions |
| Parity attempt (2-3 voting passes + verification) | ~2.4M to 3.5M total | $10.63 - $15.27 | $4.35 - $6.20 | Error floors near 10.4% at 3 votes; tens of minutes per cycle; no audit trail |
Per-viewpoint single-pass measurements: 24,824 to 24,831 input tokens, 2,500 output tokens (capped). Agentic per-viewpoint measurement across 4 turns: inputs of 10,189 / 20,463 / 28,509 / 30,315 tokens with 7,900 output tokens total. The x12 extrapolation applies a measured per-viewpoint cost to the full viewpoint set.
Why voting cannot buy determinism
There is a second structural effect: prompt caching compresses input costs on every configuration, but voting multiplies output tokens, and output tokens are never served from cache. That is why the parity attempt stays 3 to 7 times more expensive than a single pass at any caching level, while still failing reproducibility, coverage certification, and audit-trail requirements.
Measured versus modeled
- Measured: all token counts (API usage objects), the 3.6x agentic input-replay multiplier, per-cycle costs, output caps, and end-to-end wall-clock times of 35 to 105 seconds for compiled single-cycle runs versus minutes to tens of minutes for multi-pass agentic runs.
- Modeled and labeled: the x12 viewpoint extrapolation from measured per-viewpoint costs, the voting pass count (2 to 3), the majority-vote arithmetic applied to the measured flip rate, and the cached scenario (perfect within-cycle caching, the best case for the agent side since caches expire in minutes and cross-run reuse is rarely realistic).
- Corpus: fully synthetic, generated for this study at a production-realistic scale. Ratios travel to other workloads; absolute dollar figures shift with document sizes and volumes. The method is reproducible on your own data.
Frequently asked questions
How much does agentic evaluation cost per cycle?
Measured on a reproduced multi-document evaluation workload on live models: $1.34 per cycle for a single structured pass, $4.64 for a growing-context agentic run, and $10.63 to $15.27 for 2-to-3-vote configurations with verification. With realistic prompt caching those figures drop to roughly $0.64, $1.85, and $4.35 to $6.20 respectively.
Why do agentic runs cost more than single passes?
Input replay. The growing-context agent re-reads its accumulated history at every step, replaying 3.6x the input tokens of a single pass to produce the same verdicts. The waste is architectural, not model-dependent.
Does prompt caching fix agent evaluation costs?
It narrows the gap but never closes it. Cached input reads bill at roughly a tenth of the base rate, but output tokens are never served from cache, so iterative and voting configurations remain output-dominated and cost multiples of a compiled pass.
Does majority voting make agent evaluation reliable?
Partially. A roughly 20% borderline flip rate on single-pass evaluation drops to roughly 10% with 3-vote majority, at roughly 3.3x the model spend. Voting cancels randomness but converges to a borderline error floor; it cannot reach deterministic reproducibility.
Use the numbers
These measurements set the architecture multipliers, cache modeling, and quality-equalization defaults in the AI Agent ROI & TCO Calculator. Model your own volumes, labor costs, and caching assumptions there, or read the Build vs Buy guide for the decision framework.
Related pages