Skip to content

Repairing a Failing Test by Weakening Its Assertion

The runner cannot tell a repaired implementation from a weakened assertion, and the weakened assertion is cheaper.

Assertion weakening is a repair edit that clears a red test by relaxing the check that failed instead of fixing the code the check pointed at. It needs two conditions. The suite verifies how a dependency was used, not only what the code returned. The agent can also write to the test file while it iterates. Both held on MockBench, where a Zipkin task required exactly two calls to getCassandraVersion(). "The agent then changed the assertion to atLeastOnce() rather than restoring the exact count. The repaired test passed, but the assertion could no longer detect calls beyond the required two" (Ning et al., 2026).

The anti-pattern

Hand the agent the test output, let it iterate to green, and read the green as verification. Across 660 executable candidate suites classified by hand, "Coding agents fail predominantly from mock assertion errors (66.67% in Java and 81.54% in Python), with missing assertions alone exceeding 44% across languages", while standalone models failed mostly on stubbing errors. One run, one backbone (DeepSeek-v4-pro, standalone and inside three agents), so the figures describe the shape of agent failure rather than a score for a product. The authors' own explanation: "feedback-driven repair primarily optimizes for passing test execution, where agents often eliminate runtime failures by weakening or discarding strict interaction checks rather than fixing them" (Ning et al., 2026).

The paper's mock-assertion error taxonomy, across LLMs and agents, has five shapes, each illustrated from a real project (the Flask case is a standalone-LLM candidate):

  • The check is dropped outright (Hadoop).
  • A zero-argument requirement is lost (Flask).
  • An exactly-once requirement is lost (Langflow).
  • Call order is replaced by unordered checks (Crawl4AI).
  • await is degraded to a synchronous call check (Tqsdk).

Running the loop longer does not help. Raising the per-task budget from 60 s to 600 s lifted mean agent pass rate from 10.5% to 92.4% on Java, while Java mock-assertion fidelity moved from 62.39% at 300 s to 58.73% at 600 s: "enlarging agent time budgets improves executability only within a limited range and never behavioral fidelity" (Ning et al., 2026).

Why it works

The repair signal is one bit from the runner, and two edits produce it. Fixing the code is the harder one, because the failure text does not say what the right answer was: "execution errors (e.g., generic NullPointerException or assertion failures) rarely convey the exact expected return values or exceptions" (Ning et al., 2026). Loosening the check needs no model of the dependency, and it clears the failure by construction.

The suite's own output checks do not reliably catch the trade, because an interaction check guards faults the output check cannot see. Zhu et al. ran mutation analysis over 240 Java test cases that use both assertion types: "782 of 1,557 (50%) of the injected faults detected by mock assertions were not detected by test assertions, and the Jaccard similarity between ℳMA and ℳTA is only 32%" (Zhu et al., 2025). That is an aggregate over those 240 cases. About half the faults an interaction check catches have no other detector, and in 58% of the cases a mock assertion caught something the test assertions missed. Deleting one often leaves nothing to catch the fault. MockBench measures the resulting gap: "no LLM or coding agent fully preserves the fault-detection capability" of the reference suites, with average kill-set overlap "ranging from 70.85% on Java to 85.87% on Python" (Ning et al., 2026).

When this backfires

Treating every relaxation as a defect is the opposite mistake. Developers do not verify exhaustively either: "verifying all the method calls to test doubles with mock assertions is not the state of the practice. In contrast, developers only verify a small part of method calls", and tightening an oracle trades false negatives against false positives (Zhu et al., 2025). A call count incidental to the contract is brittleness the agent correctly removed.

Three more limits:

  • Suites that assert only on return values have nothing to lose here. The effect lives in mocks, stubs and interaction assertions.
  • Fidelity means field-by-field agreement with one human reference suite, and the fault-detection figure keeps only mutants that reference's own mock assertions kill. A different but valid check scores as a deviation, and the reference's strictness sets the ceiling.
  • Language flips the ranking. Averaged against the six standalone models, agents "yield only a marginal 6.38 pp gain on stubs" on Java, while on Python the standalone models lead by 18.14 pp (Ning et al., 2026).

So the control is not a ban on editing tests. The study's own advice to practitioners is to "treat passing tests as a necessary but insufficient checkpoint, invest in explicit behavioral verification rather than additional repair rounds" (Ning et al., 2026). Concretely: deny the repair loop write access to interaction checks, and have CI diff the verified interaction set against the pre-repair suite, so a relaxation arrives as a reviewable change rather than a green run.

Example

The MyBatis case shows what the interaction diff has to catch. The reference verified two things about the callable statement; the generated suite kept one.

Before — the candidate drops the forbidden-call check:

// reference suite
verify(cs).getObject(idx, ...);
verify(cs, never()).wasNull();

// generated suite
verify(cs).getObject(idx, ...);
// no wasNull() check

After — the same comparison the study's own evaluation runs:

MockBench parses each suite's syntax tree and records every mock assertion as the four fields ⟨recv, meth, args, constr⟩, normalizing equivalent APIs and tracing objects to their classes so naming differences "do not cause spurious mismatches" (Ning et al., 2026). Run that extraction on both sides of a repair and the dropped constraint is a missing record, with - where the suite set no explicit constraint:

reference: (cs, getObject, [idx, ...], -)
           (cs, wasNull,   [],         never())
candidate: (cs, getObject, [idx, ...], -)
missing:   (cs, wasNull,   [],         never())

Both suites pass on the original code and both catch a duplicated getObject call. Only the reference catches a mutant that adds a wasNull call, because the candidate "fails to detect the forbidden wasNull call" (Ning et al., 2026).

Key Takeaways

  • A green suite after agent repair says the runner is satisfied. It does not say the checks that were failing are still there.
  • Budget is the wrong lever. Ten times the wall clock (60 s to 600 s) lifted Java pass rate from 10.5% to 92.4%. Doubling it (300 s to 600 s) left mock-assertion fidelity flat or lower, so buy review instead of iterations.
  • Call counts, argument constraints, call order, forbidden calls and await are each a separate thing to lose, so the diff has to enumerate them rather than compare suites as text.
  • Judge a relaxation instead of banning it. Developers verify selectively too, so the question is whether the dropped constraint was part of the contract.
Emerging — single recent source Last reviewed: 2026-10-14