Train/test contamination

You fine-tune a model, run your benchmark, and the score jumps six points. Before you write that up, there is one question worth ten minutes: how many of those benchmark examples were in the training data? If the answer is “some”, the six points are partly a measurement of memory, not of capability — and you have no way to tell which part is which after the fact.

This is train/test contamination, also called data leakage or eval contamination. It is one of the most common and least discussed reasons an offline number fails to reproduce in production.

What contamination actually is

Contamination is any situation where information from your evaluation set is present in your training set. The clearest case is a literal duplicate: the same question, with the same answer, in both files. But it covers a wider range than most people assume:

• Exact duplicates. Byte-identical records. Usually the result of concatenating public datasets that share a common ancestor.
• Reformatted duplicates. The same example after a case change, a punctuation change, a markdown pass, or a different quoting convention. Different bytes, identical content.
• Near-duplicates. The same example with a reworded intro, a changed number, an added sentence, or a template applied. A human would call these the same item.
• Partial leakage. The eval question appears in training paired with a different answer, or the answer appears without the question. Still leakage: the model has seen the hard part.

A useful mental model: contamination is not a property of files, it is a property of examples. Two files can be disjoint at the byte level and still share most of their content.

How it gets in

Almost nobody copies their test set into training on purpose. It happens through ordinary, reasonable steps:

1. Merging public datasets. Two datasets on the Hub that look unrelated frequently share a source. Instruction-tuning collections are especially prone to this: many are recombinations of the same few seed sets.
2. Splitting after augmentation. If you paraphrase or template-expand examples and then split, variants of the same underlying item land on both sides. The split looks random and is not.
3. Re-scraping. Your eval set came from a website in March. Your training crawl hit the same site in June. Nothing about either step looks wrong.
4. Synthetic data from a model that saw the benchmark. Generate training data with a large model and you may be distilling its memorised benchmark answers into your training file.
5. Growing datasets over time. The eval set was frozen a year ago. The training set has been appended to weekly by three different people since. Nobody re-checked the overlap.

The common thread is that contamination is introduced by pipeline changes, not by a single mistake. That is why a one-off audit does not stay true, and why this belongs in CI.

Why it inflates your numbers

A model that has seen an example during training can reproduce its answer without having learned the underlying skill. On the contaminated slice, your benchmark stops measuring generalisation and starts measuring recall. The result is a score that is high, stable, reproducible — and wrong about the thing you care about.

The damage is not proportional to the contamination rate; it is worse, for three reasons:

• The effect is concentrated. If 5% of your eval set is contaminated and the model scores near-perfectly on that slice, it can lift the headline number by several points — often the same size as the improvement you are trying to demonstrate.
• It biases decisions, not just reporting. You pick hyperparameters, checkpoints and data mixes by comparing eval scores. Contamination rewards whichever run memorised more, which is usually the run that trained longer on the contaminated subset.
• It hides regressions. A genuinely worse model can score the same as a better one if it memorised more of the overlap.

The practical consequence: an uncontaminated eval set with a lower score is more useful than a contaminated one with a higher score. You cannot correct for contamination after the fact by subtracting points, because you do not know how the model would have done on those items unseen.

Three levels of overlap

Detection is usually framed as one number, but there are really three different questions with three different costs and confidences.

Exact

Compare the record text byte for byte, after joining whichever fields define an example. Implemented as a hash lookup, it is O(n) and complete: no false positives, no false negatives, nothing to tune. Always run it. If it finds anything, you have a definite problem and you can stop debating.

Normalized

Apply Unicode NFKC, lowercase, strip punctuation and symbols, collapse whitespace — then compare. This catches the same example after a reformat: smart quotes turned into straight quotes, a title-cased prompt, trailing whitespace, a markdown wrapper. Still a hash lookup, still complete, still essentially free.

In practice this level frequently finds two to five times as many matches as exact alone, because real pipelines reformat text constantly. Skipping it is the single most common reason a contamination audit reports “clean” when it is not.

Near-duplicate

The hard one. Two records are near-duplicates if they share most of their content but not all of it. The standard method:

1. Turn each text into a set of shingles — overlapping word n-grams, typically 5-grams. (For texts shorter than the n-gram size, character n-grams instead.)
2. Define similarity as the Jaccard index of the two shingle sets: the size of their intersection divided by the size of their union. Identical texts score 1.0; texts sharing nothing score 0.0.
3. Comparing every pair is O(n²) and impossible at dataset scale, so approximate it with MinHash: a fixed number of hash permutations (128 is a common choice) reduce each set to a short signature whose agreement rate estimates Jaccard.
4. Group signatures into LSH bands. Two records become candidates if any band matches exactly. Band geometry controls the similarity at which a pair is likely to surface.
5. Score each candidate on the shingle sets themselves, not on the signatures, and keep the pairs above your threshold. (Very long records are first reduced to a bottom-k sketch — the k smallest hashes — which is a uniform sample that stays stable when the record is edited.)

Note where the approximation lives: in step 4, candidate generation. Step 5 goes back to the shingle sets, so a similarity is never read off a MinHash signature — it is exact for ordinary-length records, and for very long ones it is a bottom-k estimate whose error shrinks as the sketch grows. That means near-duplicate detection can miss a small number of borderline pairs, but the similarity attached to every pair it reports is a real measurement rather than a by-product of the index.

Picking a threshold

The Jaccard threshold is the only judgement call in the whole process. Some orientation on what the numbers mean for word 5-grams:

Two adjustments worth making. If your records are short — single-sentence questions, classification labels — shift upward, because short texts have few shingles and each differing word costs a lot of similarity. If your records share heavy boilerplate — a system prompt on every row, an identical instruction header — strip the boilerplate before comparing, or you will measure the template rather than the content.

The reliable way to choose is empirical: run at 0.8, read twenty of the borderline pairs, and move the threshold based on whether you would call them the same example. It takes five minutes and beats any rule of thumb, including this one.

How to measure it

Report contamination rate as the fraction of eval records with at least one match in train. Counting records rather than pairs matters: a training file that duplicates one eval example forty times is one contaminated eval record, not forty. Pair counts tell you about the training set; record counts tell you how much of your benchmark is compromised.

Report it per level, too. “7% contaminated” is much less actionable than:

exact:      31 eval records
normalized: 12 eval records
near:       27 eval records   (threshold 0.8)
-----------------------------------------
70 of 1,000 eval records (7.00%)

The first two lines are certainties. The third depends on a threshold you chose, and a reviewer is entitled to argue with it. Keeping them separate is what makes the number defensible.

You can run all three levels on both files right now in your browser — the free checker on the home page does exactly this, locally, with nothing uploaded. For data that cannot go in a browser or for a check on every pull request, the command-line toolkit ($29, one-time) runs the same three levels offline and returns an exit code CI understands.

What to do once you find it

The instinct is to delete from the eval set. Resist it. Removing eval examples changes what your benchmark measures and breaks comparability with every number you have already published or recorded.

Clean the training set instead. Remove every training record that matches an eval record, retrain, and re-evaluate. Your eval set stays fixed, historical comparisons stay valid, and the new number is honest. The cost is a retraining run — which is exactly the cost you were going to pay anyway, later, with worse information.

Then:

• Keep the report. Store the JSON alongside the model artifacts. When someone asks in six months whether the eval was clean, you want a file, not a memory.
• Re-run the evaluation and compare. The delta between the contaminated and clean scores is the most useful diagnostic you will get all week. If it is large, contamination was driving your conclusions.
• Review the borderline pairs before deleting them. At 0.8 there will be a few pairs that are genuinely different examples that happen to share phrasing. Deleting them costs you a little training data; keeping them costs you a little eval integrity. Look at them and decide, do not guess.

Preventing it next time

• Split before you augment. Any transformation applied before the split can put variants of one item on both sides.
• Split by source, not by row, when rows come in correlated groups — same document, same user, same template. Row-level random splits leak group information.
• Freeze eval, version train. The eval set should be immutable and hashed. The training set is allowed to change, which is exactly why it needs re-checking every time it does.
• Run the check in CI. Contamination arrives with data changes, so gate data changes on it. A job that fails a pull request when the rate rises above your budget is the only version of this that stays true.
• Check every eval set you add, on the day you add it, including public benchmarks — a public benchmark is more likely to be in your training corpus, not less.

What detection will not catch

Being clear about the ceiling matters more than the pitch:

• Semantic paraphrase. Two records that mean the same thing while sharing no word 5-gram will not be found by any shingle-based method. Catching those needs embedding similarity, which is slower, threshold-sensitive in a much less interpretable way, and non-deterministic across model versions. It is a different tool with a different cost profile, and it is not obviously better — an embedding tool will also flag pairs that merely share a topic.
• Contamination through a third model. If your training data was generated by a model that had memorised the benchmark, the leaked information is in the weights, not in matching text.
• Fields you did not compare. Overlap hiding in a metadata column you excluded is invisible. Check what a “record” means in your data before you trust the number.
• Contamination in a pretraining corpus you do not control. If you fine-tune a base model, its pretraining data may already contain your benchmark. Nothing here can see that; the honest response is to say so when you report the score.

None of that makes the check less worth running. Exact and normalized overlap is extremely common, entirely detectable, and completely fixable — and the near-duplicate level catches most of what is left in practice. Finding a definite problem in ten minutes is a good trade even when the method has a known ceiling.

Check your own files in the browser → Drop a train file and an eval file on the home page. Everything runs locally in a Web Worker; nothing is uploaded. You get the per-level breakdown, a reviewable list of matched pairs, and a cleaned training file to download.