Skip to content

Benchmark Contamination as Eval Risk

Static benchmarks become unreliable as models train on their data. Decontaminated pipelines use temporal filtering and continuous fresh task sourcing to measure real capability.

The contamination problem

Models trained on large internet corpora end up seeing benchmark data. When a model has seen the test set during training, its score reflects memorization, not generalization.

SWE-rebench quantified this for coding agents: DeepSeek-V3 scores 39.7% on SWE-bench Verified but only 21.3% on decontaminated fresh tasks — an 18.4 percentage point gap attributable to contamination. GPT-4.1 shows a similar pattern: 31.1% on older tasks versus 26.7% on newer ones. [Source: SWE-rebench]

The problem extends beyond SWE-bench. LessLeak-Bench audited 83 software engineering benchmarks and found leakage ratios ranging from under 1% to 100%. StarCoder-7b achieved Pass@1 4.9x higher on leaked samples than on non-leaked samples in the APPS benchmark. [Source: LessLeak-Bench]

Teams that rely on published benchmark scores for upgrade decisions risk picking a model that memorized the test set over one that generalizes better to real-world tasks.

Decontamination mechanisms

Three mechanisms restore honest measurement.

Temporal filtering

Track the creation date of every eval task (the issue, the PR, the test) against the model's training data cutoff. Tasks created after the cutoff cannot appear in training data. SWE-rebench flags evaluations where tasks predate a model's release date, marking them as potentially contaminated on the leaderboard. [Source: SWE-rebench leaderboard methodology]

Continuous fresh task sourcing

Rather than curate a fixed benchmark once, extract tasks continuously from recent real-world activity. SWE-rebench mines merged PRs linked to resolved GitHub issues, yielding 21,336 tasks from 3,468 repositories. The pipeline runs four stages: repository filtering, LLM-driven environment setup, execution validation in isolated containers, and quality assessment. [Source: SWE-rebench]

The same principle works at team scale. Periodically add eval tasks from recent internal work to keep your suite ahead of contamination.

Standardized scaffolding

Contamination is not the only confounding variable. Differences in prompts, tools, and test-time computation inflate or deflate scores apart from model capability. SWE-rebench isolates model quality by fixing the scaffolding: identical ReAct-style prompts, 128K context window, default hyperparameters, and five runs per model with standard error reported. [Source: SWE-rebench leaderboard methodology]

Team-level defenses

You do not need to build a 21,000-task pipeline. Three practices protect against contamination at team scale:

  1. Maintain a private eval suite. Tasks drawn from your own codebase and real production incidents are unlikely to appear in any model's training data. [Source: Demystifying Evals for AI Agents]

  2. Refresh continuously. Add eval tasks from recent work — last month's merged PRs, last week's production incidents. Tasks that postdate the model's training cutoff are inherently decontaminated.

  3. Treat public benchmarks as upper bounds. Use them for directional comparison, not absolute measurement. When two models score within a few points of each other on a public benchmark, the difference may be contamination rather than capability.

When this backfires

Decontamination practices carry their own trade-offs:

  • Loss of cross-team comparability: a continuously refreshed private suite cannot be compared against other teams' or vendors' numbers. Standard public benchmarks remain the only shared yardstick for ranking models across the field.
  • Selection bias in sourced tasks: mining recent merged PRs or production incidents biases the suite toward whatever work your team happens to do. Capability regressions on task types outside that distribution go undetected.
  • Pipeline maintenance cost: running a fresh-task pipeline with isolated execution, quality filtering, and repeated runs can cost more engineering time than the measurement precision gained — especially for small teams making infrequent model decisions, for whom a lighter golden query pairs suite may suffice.
  • Apparent regressions from task drift: when the eval set changes between runs, score movements conflate model changes with task-difficulty drift. Static benchmarks at least hold the measurement rod constant.

For teams making one model-selection decision per year, triangulating two or three public benchmarks may be cheaper and nearly as informative as building a decontaminated pipeline.

Example

SWE-rebench's own leaderboard shows the pattern. DeepSeek-V3 leads on SWE-bench Verified at 39.7%, but drops to 21.3% on decontaminated tasks — a gap nearly as large as its reported score. Models with lower Verified scores show smaller drops, suggesting their original scores were less inflated by contamination. [Source: SWE-rebench]

A team using SWE-bench Verified scores alone to choose between models would rank DeepSeek-V3 highest. A team running the same models against fresh, post-cutoff tasks drawn from their own repositories would get a different ranking — one that reflects generalization to the work the team actually needs done.

Key Takeaways

  • Static benchmarks inflate scores as models train on their data — the measured gap can exceed 18 percentage points
  • Temporal filtering (using only post-cutoff tasks) is the primary decontamination mechanism
  • Continuous fresh task sourcing from real-world activity prevents benchmark stagnation
  • Private eval suites sourced from your own codebase provide inherent contamination resistance
  • Treat published benchmark scores as upper bounds, not ground truth, when making model selection decisions
Established — multiple independent sources Last reviewed: 2026-06-12
Feedback