Decontamination in CI

Who this is for

You maintain something that gets re-run: a post-training pipeline that produces a model release every few months, or a leaderboard that scores new models as they arrive. Either way, your credibility is the product. A contaminated benchmark number is not an embarrassing detail in an appendix — it is the thing people cite you for.

You almost certainly already run decontamination. Most teams in this position have an n-gram script, a list of eval sets, and someone who remembers to run it before a release. This guide is not about introducing the idea. It is about the specific ways that arrangement stops being true between one release and the next, and what to change so it does not.

Everything below is runnable today with the $29 SplitCheck toolkit or with your own scripts — the shape of the workflow matters more than whose code runs it. Where a SplitCheck flag is mentioned it is because the exact behaviour is worth being precise about, not because there is no other way to do it.

The check is not what rots — the list is

A decontamination script does not degrade. The n-gram matcher you wrote two years ago still does exactly what it did then. What degrades is everything around it:

• The eval-set list goes stale. You added three benchmarks since the list was written. Two of the paths in it now point at a moved or renamed file. Nothing errors — the script reads what it can and reports clean on the sets it actually checked, which is a subset nobody re-counted.
• The training mixture changed shape. A new source arrived with a different schema, so the field your script compares is empty for a third of the rows. Empty strings do not match anything. The contamination rate goes down, and it looks like good news.
• The threshold was tuned for the old data. A near-duplicate threshold that was right for short QA pairs is wrong for long multi-turn conversations, where a shared system prompt can carry the similarity on its own.
• The person who ran it moved teams. This is the most common one and the least technical.

Each of these produces the same symptom: a number that is still generated, still reported, and no longer measuring what its name says. The fix is not a better matcher. It is making the inputs to the check — which eval sets, which fields, which threshold — into reviewed, versioned artifacts that a pull request has to change deliberately.

What it looks like when a team publishes the numbers

Two public artifacts are worth reading before designing your own check, because between them they show both halves of the problem.

AI2’s Tulu 3 report (arXiv:2411.15124, §3.2, “Prompt Decontamination”) documents the team’s own pass over its training mixtures — which is more than most releases disclose at all. It matches on prompts only, using 8-grams; a test instance counts as overlapping when more than half its tokens have an 8-gram match against a single training instance, and a training set is called contaminated when it overlaps more than 2% of an eval set’s instances.

The instructive part is what gets reported. For Evol-CodeAlpaca against HumanEval, the decontamination table records 3.5% of the training dataset removed. A separate appendix table, measuring the other direction, puts the share of HumanEval overlapping that training set at 70.7%. Both numbers are correct and they describe the same finding. They are not interchangeable, and the distance between them is the entire argument for counting eval records rather than training records: 3.5% tells you how much data you dropped, 70.7% tells you how much of the benchmark was compromised. A pipeline that reports only the first will report a small number for a large problem.

Stanford’s HELM takes the opposite approach and is explicit about it: src/helm/benchmark/static/contamination.yaml is a hand-curated list of (model, scenario) pairs flagged as contaminated, each entry sourced to a citation in the literature rather than computed from data. That is a legitimate design — a registry of what has been reported is a different and cheaper thing than an overlap measurement — but it carries the maintenance cost this guide is about: the file has not been edited since December 2023, and the proposal to make contamination computation a first-class HELM scenario (issue #3914) was filed in November 2025 and is still open. Neither fact is a criticism of the project. They are what a curated list costs when the cadence of new models does not slow down.

Step 1 — decide what a record is

This is the decision that silently determines every number you will publish, and it is usually made by whichever field name the script happened to read first.

For instruction and preference data, there are two defensible choices:

• Compare the prompt only. An eval question that appears in training counts as contamination even if the training completion is different. This is the strict reading: the model has seen the hard part, whatever answer was attached.
• Compare prompt and completion together. The same question with a genuinely different answer is treated as a different example. This is the permissive reading, and it is the right one when your training data deliberately contains reformulations of common questions.

SplitCheck joins multiple fields with a newline and compares the result as one string, so naming both fields gives you the permissive reading and naming only the question field gives you the strict one:

# strict: an eval prompt in the training set is contamination, whatever answer follows it
splitcheck check data/train.jsonl data/evals/gsm8k.jsonl --field question

# permissive: the pair has to match
splitcheck check data/train.jsonl data/evals/gsm8k.jsonl --fields question,answer

Tulu 3 uses the strict reading — its decontamination pass matches prompts and ignores completions. That is a defensible default for post-training data, and worth copying deliberately rather than arriving at by accident.

Pick one, write it in the same file as the eval list, and treat changing it as a change to your reported numbers — because it is. A rate measured under the permissive reading is not comparable to one measured under the strict reading, and nothing in the output will warn you that the definition moved.

Step 2 — pin the eval sets in the repo

You are not checking against one eval set; you are checking against a list of them, and that list is the artifact that goes stale. Put it in the repository next to the pipeline, one path per line, and make changes to it go through review like any other code:

# evals.txt — the benchmarks this release is checked against.
# Adding a benchmark to the release means adding it here in the same PR.
data/evals/gsm8k.jsonl
data/evals/humaneval.jsonl
data/evals/truthfulqa.jsonl

Then loop over it. The important part of the script below is not the loop — it is that it distinguishes contaminated from could not run, which is the failure the stale-list problem hides:

#!/usr/bin/env bash
# check-evals.sh — one contamination check per pinned eval set.
# Exit 0 = clean, 1 = contaminated, 2 = the check itself failed.
mkdir -p reports
status=0

# The '|| [ -n "$line" ]' keeps a final line with no trailing newline.
while read -r line || [ -n "$line" ]; do
  case "$line" in ''|\#*) continue ;; esac

  if [ ! -s "$line" ]; then
    echo "::error::eval set missing or empty: $line"
    exit 2
  fi

  name=$(basename "$line" .jsonl)
  splitcheck check data/train.jsonl "$line" \
    --fields prompt,completion \
    --threshold 0.8 \
    --report "reports/$name.md" \
    --json "reports/$name.json" \
    --fail-over 0.0

  case $? in
    0) echo "clean: $name" ;;
    1) echo "::error::contaminated: $name"; status=1 ;;
    *) echo "::error::check failed to run: $name"; exit 2 ;;
  esac
done < evals.txt

exit $status

One thing that script gets wrong on purpose, to keep it readable: it uses the same --fields prompt,completion for every eval set. Real benchmarks do not share a schema — GSM8K is question/answer, HumanEval has neither — so in practice the field list belongs in the manifest next to each path, not hard-coded in the loop. Getting this wrong is the Step 1 failure arriving through the back door: a field name that does not exist in a file compares empty strings and reports clean.

Three things that script does on purpose. It fails on a missing path instead of skipping it, so a renamed benchmark cannot quietly leave the checked set. It keeps going after a contaminated eval set so one PR shows you every affected benchmark rather than the first one alphabetically. And it reserves exit 2 for its own failure: SplitCheck returns 0 at or below the threshold, 1 above it, and 2 when it could not do its job at all. Collapsing 1 and 2 into “non-zero” is the single most common way this check turns into a check that is not running.

Step 3 — run it and read the output

SplitCheck
  train: data/train.jsonl (48,120 records, jsonl)
  eval:  data/evals/truthfulqa.jsonl (817 records, jsonl)
  fields: prompt + completion
  levels: exact, normalized, near   threshold: 0.8

  exact:      12 eval record(s), 12 train record(s), 12 pair(s)
  normalized: 17 eval record(s), 17 train record(s), 17 pair(s)
  near:       23 eval record(s), 29 train record(s), 31 pair(s)

48 of 817 eval records (5.88%) also appear in the train file.

Read the headline as what it is: the fraction of eval records with at least one match in train. Counting eval records rather than pairs is what makes the number mean “how much of this benchmark is compromised”. A training file that duplicates one eval item forty times is one contaminated eval record, not forty — the pair count tells you about your training data, the record count tells you about your benchmark.

The per-level split is not decoration. The first two lines are certainties: exact and normalized matching are complete hash lookups with nothing to tune, so a non-zero count there is a fact you can act on without argument. The third line is a measurement under a threshold you chose, and a reviewer is entitled to disagree with it. Reporting one blended percentage throws that distinction away, and it is the distinction that decides whether a finding blocks a release or opens a discussion.

Step 4 — split the policy in two

The usual failure here is a single threshold that is either so strict it fires on template boilerplate until people stop reading it, or so loose it never fires at all. Split it instead: block on the certainties, report on the judgement.

# Gate: exact and normalized overlap only. Zero tolerance, blocks the merge.
splitcheck check data/train.jsonl "$evalset" \
  --fields prompt,completion \
  --levels exact,normalized \
  --fail-over 0.0 \
  --report "reports/$name.gate.md"

# Review: near-duplicates. Always exits 0; produces a report to read.
splitcheck check data/train.jsonl "$evalset" \
  --fields prompt,completion \
  --levels near --threshold 0.8 \
  --fail-over 1.0 \
  --json "reports/$name.near.json"

--fail-over is the contamination rate that is still acceptable, so 0.0 means any overlap fails and 1.0 means the job reports without ever failing. The second command is not a weaker version of the first; it answers a different question, and separating them is what lets you set zero tolerance on the part that deserves it.

Once the near-duplicate reports have been read a few times, tighten the second command into a gate of its own with a rate you can defend — --fail-over 0.005 is a common starting point — and record why that number and not another one. The reliable way to pick the threshold is still empirical: run at 0.8, read twenty borderline pairs, and move it based on whether you would call them the same example. The contamination guide has the full threshold table.

Step 5 — make it a required check

A check that runs is not the same as a check that blocks. Two details decide whether this survives a deadline.

Trigger on the data, not on everything. Contamination arrives with data changes, so the workflow should run when the training data, the eval sets, or the eval list itself change:

name: Contamination check
on:
  pull_request:
    paths:
      - "data/train/**"
      - "data/evals/**"
      - "evals.txt"

jobs:
  splitcheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install ./toolkit
      - run: bash check-evals.sh
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: contamination-reports
          path: reports/

if: always() matters more than it looks. Without it the reports are not uploaded on exactly the runs where you want to read them.

Then watch out for the skipped-but-required trap. If you mark this job as a required status check in branch protection while it has a paths filter, every pull request that does not touch those paths will sit forever waiting for a check that will never report. The standard fix is a second workflow with a job of the same name, triggered on the inverse paths-ignore filter, that does nothing and succeeds. It looks redundant and it is the difference between a required check and a permanently blocked queue.

Step 6 — keep the rate comparable

For a team releasing on a cadence, the absolute contamination rate is much less interesting than its movement. One number in isolation invites arguing about the threshold. Two numbers from consecutive releases, measured identically, do not.

The JSON report is the artifact to keep — one per eval set, per release, stored with the model artifacts rather than in a CI log that expires:

jq '{
  eval: .eval.name,
  rate: .contamination_rate,
  records: .contaminated_eval_records,
  exact: .counts.exact.eval_records,
  normalized: .counts.normalized.eval_records,
  near: .counts.near.eval_records,
  threshold: .options.threshold,
  fields: .options.fields,
  truncated: .run_truncated
}' reports/truthfulqa.json

Keep threshold and fields in the row, not just the rate. They are what makes two releases comparable, and they are exactly what changes without anyone announcing it. A rate that dropped because someone widened the field list is not an improvement, and six months later the only way to know which happened is that you wrote it down.

What breaks at scale

At the sizes this audience works with, two failure modes matter more than anything in the method itself.

Truncated runs report lower bounds. Near-duplicate detection generates candidate pairs before scoring them, and any implementation has to cap that generation somewhere or a pathological mixture will exhaust memory. SplitCheck stops at a candidate-pair ceiling and sets run_truncated: true in the JSON report when it does. When that flag is set every count is a lower bound and the cleaned training file is incomplete — the number is not wrong, it is a floor, and reporting a floor as a rate is how a bad release gets signed off. Assert on the flag in CI:

jq -e 'if .run_truncated then
  ("detection truncated - counts are lower bounds\n" | halt_error(2))
else . end' reports/*.json > /dev/null

Boilerplate becomes the signal. A system prompt on every row, an identical instruction header, a shared answer template — at near-duplicate thresholds these dominate the similarity and you end up measuring the template rather than the content. Strip the shared prefix before comparing, or compare a field that does not contain it. If a near-duplicate run suddenly flags a large fraction of a benchmark, check for this before you check for contamination; it is the more likely explanation.

What this will not catch

Stating the ceiling matters more here than anywhere else, because this audience will find it anyway:

• Rewritten and translated benchmark items. Lexical overlap methods — n-gram, MinHash, anything shingle-based — compare surface text. A benchmark item rewritten thoroughly enough to share no word 5-gram with the original will not be found, and no choice of level or threshold changes that. The published demonstration is blunt: in “Rethinking Benchmark and Contamination for Language Models with Rephrased Samples” (arXiv:2311.04850), a Llama-2-13B trained on a rephrased MMLU test set reached 85.9 on MMLU while remaining undetectable by n-gram overlap, and CodeLlama models trained on rephrased samples went from 32.9 to 67.7 (7B) and 36.0 to 81.1 (13B) on HumanEval — against GPT-4’s 67.0 on the same benchmark. Catching that class needs embedding search or an LLM judge over the top candidates, which is what the authors built alongside the paper.
  So: a clean report is not proof your eval is clean. It is proof that one detectable class of contamination is absent, which is worth having and is not the same claim.
• Contamination inherited from the base model. If you post-train someone else’s base model, its pretraining corpus may already contain your benchmark. Nothing in your pipeline can see that. The honest response is to say so when you report the score.
• Leakage through a generator model. Synthetic training data produced by a model that memorised the benchmark carries the leak in the weights, not in matching text.
• Fields you did not compare. Overlap hiding in a metadata column you excluded is invisible. This is the same problem as Step 1, arriving later.

None of that argues against running the check. Exact and normalized overlap is common, entirely detectable, and completely fixable; near-duplicate detection catches most of what is left in practice. The argument is only against reporting “decontaminated” as though it were a binary that has been achieved.

What this costs

Everything in this walkthrough runs today with the $29 SplitCheck toolkit — a Python package, no service, no account, no data leaving your infrastructure — or with your own scripts, if you would rather own the matcher. If your data changes rarely, that is very likely the right amount of tooling, and this page has already given you the workflow for free.

What the toolkit does not do is enforce anything. It cannot stop someone editing evals.txt to drop a failing benchmark, it has no notion of an approved exception so a known false positive re-fails every PR forever, and it keeps no history across releases — the JSON files are yours to store and diff.

Disclosure: this page was written with AI assistance and reviewed before publication. Its command-line flags, exit codes and JSON field names were checked against the SplitCheck toolkit source they describe. If you find something here that is wrong, tell us and we will correct it on the page rather than quietly.