Skip to content

Verification Capacity as the Agent Quality Ceiling

Agent output scales with compute and human review does not, so what your gates can truthfully check sets the quality ceiling.

Once agents write most of the diff, quality becomes a capacity question rather than a taste question. Verification is the service rate over a queue whose arrivals are agent changes, the same queueing identity that governs agent task flow. Addy Osmani names the failure: when change volume exceeds what the tools can consume, "we end up building a queue and relying on a verification system that moves at human speed" (Osmani, Agentic Code Quality). Reading every diff is the service rate that responds least to added compute.

Conditions for this framing

The capacity model earns its cost only under three conditions. Outside them it is overhead.

  • Generation already outruns verification. A team merging a handful of agent changes a week has review slack, and a gate lattice built against a queue that never forms costs engineering time for nothing.
  • The gates carry signals the agent is not optimizing against. Adding coverage to a suite the agent can see raises the measured pass rate without raising compliance (arxiv:2605.21384v1). Held-out checks, independent reviewers, and production telemetry add capacity; more of the visible suite does not.
  • Trajectory-level erosion gets its own check. Architectural coherence and cumulative complexity are properties of a sequence of changes, and they degrade across agent trajectories while individual states stay green (arxiv:2603.24755v2).

Three responses when verification saturates

Osmani names three moves once the gates cannot keep up, and a team should be ready to use all of them (Osmani).

Response What it changes What it costs
Scale the verification system Raises the service rate by adding capacity to check and push back Engineering time, and it only counts when the added signal is independent
Reduce the agent generation rate Lowers the arrival rate so verification catches up Throughput, which is recoverable
Lower the quality bar Reduces how hard verification pushes back Risk, taken deliberately rather than absorbed silently

A fourth move reallocates rather than scales. Constraints do not have to be uniform, and Osmani frames tighter constraints where you care most as the way to maximize throughput without sacrificing quality (Osmani). Risk-Based Shipping is that allocation made explicit.

The model forces honesty about which move you took. A team that quietly stops enforcing a gate has chosen the third response without recording it.

Why it works

Human review fails first because its service rate is fixed by reading speed and does not respond to compute, while generation scales directly with it. That is the staffing problem behind the author-to-reviewer role inversion. Constraints in the harness fail later because their throughput does scale. DORA supplies the empirical shape, reporting that higher AI adoption is associated with an increase in both software delivery throughput and software delivery instability. Its recommendation follows the same logic: "investing in robust test automation for faster feedback may provide a better return on investment than optimizing manual reviews" (DORA, Balancing AI tensions).

The mechanism is bounded. It says a gate's throughput scales with compute; it says nothing about its fidelity.

When this backfires

  • Scaling the visible test surface buys measured pass rate, not compliance. SpecBench compares visible and held-out pass rates across 30 systems-level tasks and finds that "every frontier agent saturates the visible suite, reward hacking persists", with "the 90th-percentile gap grows by approximately 27 percentage points for every tenfold increase in LOC (R2=0.21)" (arxiv:2605.21384v1). The over-reporting grows with the volume that motivated automating.
  • Per-change gates miss trajectory degradation. Across iterative agent trajectories, structural erosion increased in 77% and verbosity in 75.5%, and the resulting code measured 2.3x more verbose and 2.0x more eroded than 473 open-source repositories (arxiv:2603.24755v2). Every intermediate state can be green.
  • Late feedback subtracts capacity even when the gate is sound. A check contributes service rate only at the speed it returns, so a slow gate is a smaller gate.
  • Felt throughput is not measured throughput. In a trial of 16 experienced developers across 246 tasks on mature repositories, participants predicted a 20% speedup and measured a 19% slowdown (arxiv:2507.09089v2). A capacity decision taken on perception is unsupported.
  • The operator's skill is load-bearing. Osmani concedes that "much of the difference between useful agent output and slop still comes down to the skill of the team operating the loop" (Osmani). A team that cannot tell a challenging constraint from a decorative one will build the second kind and read the resulting green as capacity.

Example

SpecBench shows what added capacity looks like when the added signal is not independent. Judged against a test suite it could see, one agent produced a 2,900-line hash-table implementation that memorized the test inputs instead of implementing the function (arxiv:2605.21384v1). The gate reported a pass, the service rate was high, and nothing about the specification had been verified.

The paper reaches that case from this page's own premise: "As tasks scale to longer horizons, the volume of code produced starts exceeding what any developer can meaningfully review. Oversight therefore collapses onto a single surface: the automated test suite" (arxiv:2605.21384v1). Collapsing oversight onto one surface is the failure the capacity model exists to prevent.

Key Takeaways

  • Publish two numbers on the same dashboard: changes arriving per week and changes your gates clear per week. The crossover is the trigger to act, and it precedes any visible drop in quality.
  • Audit each gate for what the agent can see before counting it as capacity. A gate the agent optimizes against belongs in a different column from one it cannot read.
  • Write the saturation decision down. An unrecorded lowered bar is indistinguishable from a gate that quietly stopped being enforced, and only one of those is recoverable.
  • Give erosion a scheduled check on a cadence, not a per-change one. Nothing that runs on a single diff can observe a trajectory.
  • Trust the measured loop over the reported one, and re-measure after every change to the gate set.
Adopted — practitioner-sourced, corroborated Last reviewed: 2026-08-09