Skip to content

Held-Out Tasks as a Harness Shortcut Defense

Held-out tasks change the questions and leave the protocol alone, so a shortcut living in the protocol survives the swap.

Swapping in fresh tasks moves one variable. A benchmark also carries a protocol: "file names, directory layout, metadata, tool aliases, demonstration order, feedback format" (Zhu et al., 2026, §3.1). Held-out evaluation resamples the tasks and holds that fixed, so "tasks change, but the protocol correlation does not" (§1).

When the gap is real

Three conditions have to hold together. A team missing one of them can keep using held-out tasks.

  • A search procedure, not a person, proposes harness edits from repeated benchmark feedback. That is the pressure that finds a benchmark-wide shortcut.
  • The protocol carries one. "58.1% of questions in the benchmark OfficeQA Full mention numerical scales such as millions or billions", and in its 697-document corpus "the next nonblank line after 95.2% of unit statements begins a table" (Zhu et al., 2026, §1).
  • Production does not share that protocol. Where the deployed agent sees the benchmark's layout and tool aliases, protocol-dependent gain is gain.

What the held-out score misses

Vary the protocol as well, under checks that pin the question, answer, documents, access and scorer. Then measure how much of the tuned harness's gain over the starting harness disappears. In the first round of CHASE (Counterfactual Harness Search and Evolution), the Challenger re-encoded retrieved text as table context. On the evolution split that moved the evolved harness's gain over its starting point from 8.16% on the released benchmark to −5.10% on the counterfactual, with question, answer and documents untouched (Zhu et al., 2026, §5.3). The authors read the reversal as a gain that "depends on how retrieved evidence is represented", and a task swap leaves that representation unchanged.

The gain can also run the other way, which I did not expect. On the 208 Syn-Ledger certification tasks, all three optimized methods gained more over the starting harness under the shortcut-neutralized benchmark than under the released one: for CHASE, 23.08% against 5.29%. Absolute scores still fell, for CHASE from 87.18% to 38.30% (Zhu et al., 2026, Table 2).

Why it works

A validity-preserving protocol change moves the one variable held-out tasks cannot, so it separates capability from protocol dependence. The paper names what the difference measures: Δ_BS is "the extra gain obtained by the 'bad genius' Proposer through increased reliance on the benchmark-wide shortcut rather than improved task-solving capability" (Zhu et al., 2026, §3.1). Seven invariants gate each transformation and every one must pass (Table B.2).

When this backfires

  • The tuning loop may not be worth defending. Compared against test-time scaling and discovery baselines "under comparable feedback and inference budgets", "automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization" on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6 (Wang et al., 2026).
  • Search cost is the headline. Against HarnessCompass, the fixed-gate baseline, CHASE "runs 2.47 times as many attempts and uses 2.43 times as many total tokens" during search (Zhu et al., 2026, §C.5).
  • A constraint can consume a whole round. After the first counterfactual entered the archive, every second-round candidate violated the ε=0.05 constraint despite released gains as high as 15.31%, so the method fell back to the original harness (Zhu et al., 2026, §5.3).
  • A transformation that is not provably semantics-preserving moves the problem instead of solving it. Writing about LLM reasoning tests rather than agent harnesses, the LGMT authors say informal transformations "frequently introduce semantic drift". Developers then struggle to tell a genuine defect from "a follow-up test case that inadvertently altered the original underlying logic" (Zhou et al., 2026, §1).
  • A clean archive reads stronger than it is. The authors scope their result to "the evaluated counterfactual family, whose coverage is limited by our three-round search budget" (Zhu et al., 2026, §6).

Key Takeaways

  • Held-out tasks resample the semantic task and leave the benchmark protocol fixed, so a protocol-level shortcut survives them.
  • Measure how much of the tuned harness's gain disappears under a validity-preserving protocol change, not the score change, because the change moves the baseline harness too.
  • One counterfactual turned a reported 8.16% gain into −5.10% with questions, answers and documents unchanged.
  • The released gain can understate too: CHASE's 5.29% released gain was 23.08% once the shortcut was neutralized, even as its absolute score fell.
  • Price it first. The counterfactual search ran 2.47 times the attempts of the fixed-gate baseline, on evidence of one real benchmark and one synthetic at a three-round budget.
Emerging — single recent source Last reviewed: 2026-09-19