Skip to content

Code Health as a Signal for Agent-Generated Test Quality

Code health of the file under test shifts how good an agent's generated tests are, and line coverage barely registers the shift.

Code health of the file under test weakly predicts how effective an agent's generated unit tests will be. Across 24,620 competitive-programming files in Python, Java and C++, CodeScene's CodeHealth score correlated with the mutation score of tests generated by Qwen3-Coder-30B at ρ = +0.18, +0.21 and +0.34 respectively, and with line coverage at only +0.11, +0.01 and +0.15 (arXiv:2608.18645v1). The association strengthens as the metric becomes harder to satisfy, so a team judging agent-written tests by coverage alone sees almost none of it.

Where the signal holds

The result is narrow. Four conditions bound it.

  • You measure with mutation score. For Java, the study's best-covered language, code health moved line coverage by ρ = +0.01, falling to −0.02 once source lines were controlled for (arXiv:2608.18645v1). Mutation score is the instrument that registers the effect, for the reasons in Mutation Testing as a Quality Gate.
  • The code under test is assumed correct. A 2026 replicability study found that coverage and mutation "can provide meaningful signals when comparing across models" in regression-style settings where the code is reasonably assumed bug-free, and that "in another common scenario where the code-under-test may already be buggy and the goal is to expose the bug within the code-under-test, they no longer serve as reliable indicators" (arXiv:2607.22880v1). Writing characterization tests around a suspected defect falls in the second case.
  • The language spends tokens on structure. For Python, the token association "essentially disappears" once source lines are controlled for, and the mutation-score correlation stops being statistically significant, on a subset of just 251 files (arXiv:2608.18645v1).
  • Generation is single-shot. The study issued one prompt per file with no execution feedback and no repair loop. An agent that runs the suite and iterates has a correction mechanism this experiment withheld.

What the token numbers support

Lower-health code costs more to read. In the study's lowest code-health band, median input tokens ran 45.1% higher for Java and 43.8% higher for C++ than in the near-perfect band (arXiv:2608.18645v1). Those medians compare different files rather than the same functionality written twice, so file length and content sit inside the gap. The controlled version uses matched repository pairs sharing architecture, dependencies and external behavior, and puts the saving at 7 to 8% with no change in pass rate (arXiv:2605.20049v1). Size a refactoring business case off the smaller figure. Code Cleanliness as an Agent Cost Lever carries that argument in full, and Language Choice as an Agent Token-Cost Lever covers the between-language version.

Why it works

Executing a line and killing a mutant demand different things from the model. Coverage rises when the generated test reaches a statement. Mutation score rises only when the test asserts something specific enough to fail after that statement changes. The paper's own reading is that "mutation score captures whether tests detect injected behavioral changes, whereas coverage only captures whether code elements are executed", and that ρ climbs "as the metric becomes more semantically demanding" (arXiv:2608.18645v1). CodeHealth aggregates local properties such as nesting and complexity (arXiv:2608.18645v1), and on that reading those properties cost the model the understanding a discriminating assertion needs while leaving it able to reach the line. The token side has no equivalent explanation. The authors offer subword fragmentation of camelCase identifiers, chained calls and generic types as a hypothesis for Java, and label it as one.

When this backfires

  • Budgeting a refactor against the 45% figure. A payback calculation built on it overstates the controlled return by roughly six times.
  • Python codebases, or anywhere low health mostly means more lines. The paper attributes the Python token cost mainly to source lines, so restructuring adds little on top of deleting them (arXiv:2608.18645v1).
  • Reading the C++ result as a strong endorsement. C++ produced the largest correlation and the worst tests. Median line coverage ran from 3.2% in the lowest band to 35.2% in the highest, so much of that correlation tracks whether the generated test compiled and ran at all.
  • Treating CodeHealth as a neutral instrument. Two of the paper's four authors give CodeScene as an affiliation, and CodeScene owns the metric.
  • Generalizing past competitive programming. The corpus is standalone CodeContests solutions rather than conventional production units, which the authors name as the main threat to external validity. One 30.5-billion-parameter open-weight model generated every test.

Key Takeaways

  • Code health of the file under test is a real but weak predictor of agent-generated test quality, at ρ = +0.18 to +0.34 on mutation score
  • Line coverage does not register the effect; on Java it is flat at ρ = +0.01, so a coverage-based gate reports a refactor as having done nothing
  • The 43.8 to 45.1% token gap compares different files; the matched-pair estimate for holding functionality fixed is 7 to 8%
  • The signal is scoped to regression-style generation over code assumed correct, in languages where poor health means more than extra lines
  • Provenance limits the claim: one open-weight model, a competitive-programming corpus, and a metric owned by the authors' employer
Emerging — single recent source Last reviewed: 2026-08-23