Skip to content

Tunable Effort Levels for Code Review Agents

Expose review depth as a per-PR dial backed by a published bug-discovery curve, so reviewers and routing policies trade thoroughness against cost where it matters.

The primitive

A tunable code-review agent ships with named effort levels. Each level pins a point on a published bug-discovery curve. A reviewer or routing policy picks one per PR.

Cursor's Bugbot externalized this dial on 2026-05-11:

Level Bug-discovery rate Reviewer-stated trade-off
Default 0.7 bugs/run Optimized for efficiency and speed
High 0.95 bugs/run "More reasoning time; more expensive and slower, but finds more bugs"
Custom Operator-defined Natural-language policy: "Describe when Bugbot should use default or high effort"

High finds 35% more bugs at constant 80% resolution rate. Developers address the extra flags at merge time rather than silently dismissing them.

This is the per-PR analogue of heuristic effort scaling (agent-tiered per query) and interactive effort sliders (operator-tiered per turn). Here the unit is one PR, and the decision sits with a reviewer or routing policy.

Why a dial, not a constant

Single-calibration agents force a compromise. Calibrate for thoroughness and routine PRs drown in commentary: developers override more than 30% of flags until the tool is functionally disabled. Calibrate for signal and the agent misses real regressions in high-stakes code.

GitHub Copilot's review began as the implicit binary form: in 29% of reviews the agent stays silent and in 71% it surfaces actionable feedback. A multi-level dial generalizes it: silence is the lowest rung, full agentic exploration the highest. Copilot externalized that dial in a preview on 2026-06-02 and took it to general availability on 2026-08-07, renaming the preview low and medium tiers to Lite and Balanced. Lite is the default standard review; Balanced routes complex logic, security-sensitive code, and cross-service changes to a higher-reasoning model (GitHub Docs).

That release also settled where the dial sits. An organization default applies across its repositories, a repository can override it, and whoever requests a review can pick a level for that run without changing either default. Both vendors now expose a per-PR dial; what differs is who turns it. Cursor's Custom level is a policy the tool evaluates automatically, while Copilot's is a human decision at request time.

Calibration is the pattern

Effort labels are hedge words without a published curve.

Cursor publishes two metrics:

  • Bug-discovery rate — bugs found per run, measured on BugBench, a curated benchmark of real diffs with human-annotated bugs (Building Bugbot)
  • Resolution rate — share of flagged bugs authors address at merge, classified by an LLM-as-judge validated against humans

Holding resolution rate constant across levels is the calibration commitment: High costs more but does not flood with low-quality flags. Resolution rate gates whether the higher-effort flags are useful.

Copilot publishes no equivalent curve. Lite and Balanced are documented by description rather than by a measured bug-discovery rate, so the tiers shipped as an interface without the evidence that makes a dial actionable. A routing policy tuned against Cursor's numbers has no basis on Copilot.

Routing axes

The dial composes with structured routing across four axes:

graph TD
    PR[Pull Request] --> Router{Routing policy}
    Router -->|Critical path<br/>auth, payments, crypto| High[High effort]
    Router -->|Large changeset<br/>1000+ LOC| High
    Router -->|Low-trust author<br/>new contributor| High
    Router -->|High historical defect rate<br/>in touched dirs| High
    Router -->|Otherwise| Default[Default effort]
  • File-path criticality — auth, payment, or crypto paths route to High. This is the same axis tiered code review uses for human escalation.
  • Change size — diffs above an LOC threshold route to High.
  • Author trust — first-time contributors or external forks route to High. This is the same signal CODEOWNERS already encodes.
  • Historical defect rate — directories with elevated post-merge bug rates route to High. Cursor exposes no signal, so encode it in Custom.

Cursor's Custom level encodes this as a natural-language policy evaluated per PR.

Cost-performance tie

High-effort review on critical paths is cheaper than post-merge remediation. Cursor's per-run pricing is $1.00 to $1.50 by PR size, and effort levels require usage-based billing.

The arithmetic only holds when routing concentrates High on high-stakes runs. A reviewer toggling High on every PR pays the premium and accumulates the alert fatigue the dial existed to avoid.

When this backfires

  • Resolution rate is not precision. It counts both "developer fixed it" and "developer made it go away." The 35%-more-bugs claim at High does not rule out more false positives that authors muted. Teams needing a precision floor (security under 3% FPR, style under 2%) measure noise themselves.
  • Routing policy can drift in Custom. Natural-language Custom is an instruction file with the same primacy and drift issues. A correct policy in May 2026 may misroute six months later, and no eval is exposed by default.
  • High-effort defaults create alert fatigue. Signal over volume is the dominant trust factor. Pinning High at the policy level burns attention on routine PRs and un-funds the high-stakes runs the dial protects.
  • Cost-blind defaults backfire. Without per-PR cost ranges, reviewers toggle High on everything, collapsing the dial back to a fixed-pipeline calibration.
  • No vendor provides author or defect-rate signals. Cursor exposes path-based routing but not author-trust or directory-defect-rate routing, so teams encode those manually in Custom. Copilot labels which effort level ran in the timeline and the pull request overview comment (GitHub, 2026-08-07), which records the outcome without recording the axis that triggered it.

Tunable effort assumes a published curve and a routing policy that concentrates the higher tier on the runs that pay for it. Without both, the dial reduces to per-PR cost optionality with no signal benefit.

Example

A monorepo with mixed-risk paths uses a Custom policy to concentrate High effort:

Use High effort when:
- The PR touches src/auth/, src/payments/, or src/crypto/
- The diff exceeds 500 lines
- The author has fewer than 10 merged PRs in this repo
- More than half the changed files have had a post-incident commit in the last 90 days

Use Default effort otherwise.

A 30-line fix in tests/integration/ from a long-tenured engineer runs at Default (0.7 bugs/run), fast, and silence on style reads as real silence. A 1,200-line refactor touching src/auth/session.ts from a first-time contributor runs at High (0.95 bugs/run), and the reviewer reads each flag knowing the routing concentrated effort here on purpose.

Pinning Default at the team level and letting Custom escalate keeps escalation policy-driven rather than manual: it preserves the silence-as-output contract for routine work while making the higher tier opt-in by policy, not per-PR clicking.

Key Takeaways

  • Bind the dial to one PR and give the decision to whoever routes it. That is what separates this from heuristic effort scaling and interactive effort sliders, where the unit is the agent or the query rather than the change under review.
  • Before adopting a dial, ask the vendor for the bug-discovery curve behind each level. A level with no published rate is a hedge word, not a knob.
  • Resolution rate answers "did the flag get addressed," not "was the flag correct." A dismissed flag and a fixed one count the same. Check which question a vendor's number answers, and measure noise yourself if the review needs a precision floor.
  • Routing policies concentrate High on the axes that pay for it — file-path criticality, change size, author trust, historical defect rate — the heuristic siblings of risk-score threshold calibration. Before pinning High globally, check whether you can still name the axis that triggered each escalation; if not, the routing has already collapsed into a fixed-pipeline calibration.
  • Custom natural-language policies are themselves drift-prone instruction surfaces. Treat them like CLAUDE.md: review periodically, gate changes through eval.
Established — multiple independent sources Last reviewed: 2026-08-08