Benchmark contamination scoreboard

Each row below compares one public training set with one benchmark’s test split, using the same detection code as SplitCheck’s free browser checker (the settings are under Method). Only the prompt side is compared: the benchmark’s prompt field (question or prompt) against the training set’s prompt field (question, query, instruction or problem, plus input or context where the set has one). Each pair’s table lists the exact fields. A match means overlapping text in a published dataset. It says nothing about any particular model’s scores. This page does not reproduce benchmark items; matches are listed by identifier.

Summary

Every count in this table is benchmark items, not pairs. “Any level” counts items with at least one matching training record. The three level columns count items with at least one match at that level. One matched pair (a benchmark item and a training record) is counted only at its strongest level. But one item can match different training records at different levels, so the level columns can add up to more than “any level”.

Each figure belongs to its own pair. The rows are not added together anywhere on this page, because one benchmark item can match in several training sets. Every run completed within its candidate-pair budget, so no count here is a truncated lower bound.

Results by pair

For each pair: the pinned files, how many records were compared, what text was compared, the license the publisher states, the settings, and up to 5 example matches. Examples are picked so that, where possible, every level with a match appears, with one example per benchmark item. Near-duplicate examples alternate between the highest and the lowest similarity found.

GSM8K train vs GSM8K test

None of the 1,319 GSM8K test items matched any of the 7,473 GSM8K train records at any level.

Same-repository pair: GSM8K's own train and test splits, published together by the benchmark's authors in the same repository.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 5.0 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

MetaMathQA vs GSM8K test

None of the 1,319 GSM8K test items matched any of the 395,000 MetaMathQA records at any level.

The dataset card at the pinned revision states that all MetaMathQA data is augmented from the GSM8K and MATH training sets and that none of it is from the test sets.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 202.3 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

• Detector note: 4 train record(s) had empty text for the selected field(s) and were skipped.

Orca-Math 200k vs GSM8K test

None of the 1,319 GSM8K test items matched any of the 200,035 Orca-Math 200k records at any level.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 114.3 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

Stanford Alpaca vs GSM8K test

None of the 1,319 GSM8K test items matched any of the 52,002 Stanford Alpaca records at any level.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 8.3 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

Stanford Alpaca vs MMLU test

2 of 14,042 MMLU test items (0.01%) match at least one Stanford Alpaca record. By level: 2 items with an exact match, 0 with a normalized match and 0 with a near-duplicate match. On the training side, 1 of 52,002 records matches at least one MMLU test item (2 matched pairs in total).

Every matched MMLU test item is listed below, as identifiers so you can look each one up in the pinned files yourself. The word counts are the length of the compared text on each side (whitespace-separated words). The matched MMLU test texts are 6 words long.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 15.8 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

Databricks Dolly 15k vs MMLU test

2 of 14,042 MMLU test items (0.01%) match at least one Databricks Dolly 15k record. By level: 1 item with an exact match, 1 with a normalized match and 0 with a near-duplicate match. On the training side, 10 of 15,011 records match at least one MMLU test item (10 matched pairs in total).

Every matched MMLU test item is listed below, as identifiers so you can look each one up in the pinned files yourself. The word counts are the length of the compared text on each side (whitespace-separated words). The matched MMLU test texts are between 5 and 7 words long.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 20.4 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

Open-Platypus vs MMLU test

None of the 14,042 MMLU test items matched any of the 24,926 Open-Platypus records at any level.

The dataset card at the pinned revision, under "Data Contamination Check", says its publishers removed approximately 200 questions that appear in the Hugging Face benchmark test sets.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 37.3 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

Code Alpaca 20k vs HumanEval

None of the 164 HumanEval items matched any of the 20,022 Code Alpaca 20k records at any level.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 3.9 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

evol-codealpaca-v1 vs HumanEval

1 of 164 HumanEval items (0.61%) matches at least one evol-codealpaca-v1 record. By level: 0 items with an exact match, 0 with a normalized match and 1 with a near-duplicate match. On the training side, 1 of 111,272 records matches at least one HumanEval item (1 matched pair in total).

Every matched HumanEval item is listed below, as identifiers so you can look each one up in the pinned files yourself. The word counts are the length of the compared text on each side (whitespace-separated words). The matched HumanEval texts are 43 words long.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 133.1 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

Magicoder OSS-Instruct 75K vs HumanEval

None of the 164 HumanEval items matched any of the 75,197 Magicoder OSS-Instruct 75K records at any level.

Its publishers name this file data-oss_instruct-decontaminated.jsonl. The dataset card at the pinned revision does not describe what was removed.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 174.3 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

Positive control

The same code and settings were also run on a pair where overlap is expected, because the training set was built from the eval-side set (see the dataset card note below), and kept out of the summary table above. It shows that the detector finds prompt overlap when one set was built from another; it says nothing about contamination of any benchmark’s test split.

MetaMathQA vs GSM8K train

7,182 of 7,473 GSM8K train items (96.11%) match at least one MetaMathQA record. By level: 7,178 items with an exact match, 0 with a normalized match and 46 with a near-duplicate match. On the training side, 80,221 of 395,000 records match at least one GSM8K train item (80,245 matched pairs in total).

The dataset card at the pinned revision states that all MetaMathQA data is augmented from the GSM8K and MATH training sets and that none of it is from the test sets.

Example matches (5 of 7,182 matched items), as identifiers so you can look each one up in the pinned files yourself. The word counts are the length of the compared text on each side (whitespace-separated words). The detector's list of matched pairs for this run was capped at 5,000: it keeps each matched item's strongest match, in item order, until the cap. The examples come from that list, so a level with matches can be missing from them.

Settings: levels exact, normalized, near; threshold 0.8; 128 permutations in 12 bands of 10 rows; word 5-grams (character 8-grams for short texts); sketch cap 512; candidate-pair budget 2,000,000 (default). Detection took 205.6 s on the machine that produced this page. Run 22 September 2026, detection core 1.0.0.

• Detector note: 4 train record(s) had empty text for the selected field(s) and were skipped.

Method (as run on 22 September 2026)

Same code as the checker. The numbers come from detectContamination in SplitCheck’s detection core, version 1.0.0. That is the same detection code the free checker on the home page runs in your browser, and these runs used the checker’s default detection settings (levels, threshold, permutations, shingles). The checker picks text fields automatically; here the compared fields were set per dataset (listed in each pair’s table), because the two files name their prompt fields differently. A script downloads each pinned file, checks its SHA-256, turns every row into one text field (the “compared text” in each pair’s table), runs the detector and writes the results to a JSON file. This page is built from that file. The script lives at apps/splitcheck/scripts/scoreboard in SplitCheck’s repository, which is not public yet, so you cannot run it yourself today. What you can check is every pinned file (links and SHA-256 hashes above) and every example, by its identifiers.

Whole files, no sampling. Every row of every file was compared. Rows whose compared text was empty are skipped by the detector, and each pair’s table gives their count.

Exact. The two compared texts are identical, character for character, after leading and trailing whitespace is trimmed. This is a hash lookup, so it has no false positives.

Normalized. Both texts are put through Unicode NFKC normalisation and lowercased. Every punctuation and symbol character (Unicode categories P and S) is replaced by a space, runs of whitespace become one space, and the result is trimmed. The normalized texts are then compared exactly, again as a hash lookup. A pair that is already an exact match is not counted again here.

Both lookups find every match at their level only when a run does not stop at its candidate-pair budget (see the work budget below), because exact and normalized hits are charged against that budget too. No run on this page stopped at its budget, so the exact and normalized counts here are complete.

Near-duplicate. Each normalized text becomes a set of word 5-grams. A text shorter than 5 words uses character 8-grams instead, and a text shorter than that is one single shingle. Each shingle is hashed to 32 bits, and a record with more than 512 distinct shingles keeps its 512 smallest hashes. MinHash signatures with 128 permutations, split into 12 bands of 10 rows, pick the candidate pairs. Every candidate is then scored on its two shingle sets, never on the signatures. The score is the Jaccard similarity: exact when the two records have at most 512 distinct shingles between them, and the standard bottom-k estimate over the 512 smallest hashes otherwise. Candidates scoring at least 0.8 count as near-duplicates.

The banding only decides which pairs get scored, and it can miss real matches. By the standard LSH formula, 1 − (1 − s¹⁰)¹², the chance that a pair with similarity s becomes a candidate at all is 74% at s = 0.8, 93% at s = 0.85, 99% at s = 0.9 and over 99% at s = 0.95. The near-duplicate counts are therefore lower bounds, most of all for pairs close to the threshold.

Work budget. The detector stops after examining a fixed number of candidate pairs (2,000,000 by default) and then marks the run as incomplete. A run that stops there is re-run with a larger budget, and any pair whose budget was raised shows it in its settings line.

What these numbers do and do not say

• The three levels mean different things. An exact match is the same text. A normalized match is the same text once case, punctuation, symbols and spacing are ignored. A near-duplicate is a similar text, above a threshold we chose.
• Normalisation can merge items that are not the same. It removes arithmetic operators, comparison signs, minus signs, currency and percent signs, and decimal points, and it turns superscripts and fractions like ½ into plain digits. So “3 + 4” and “3 − 4” normalise to the same string, as do “−7” and “7”, or “1.5” and “1 5”. For math items especially, check a normalized match by hand before calling it a copy.
• A near match at 0.8 can be two different items. Short questions written from the same template share most of their word 5-grams and can score above the threshold while asking different things. That is why every example is given as identifiers you can look up.
• The near-duplicate level can also miss real matches. Candidate search is probabilistic (see Method), so near-duplicate counts are lower bounds. Exact and normalized counts are complete here, because no run on this page stopped at its candidate-pair budget.
• Jaccard similarity is computed over the whole record. A benchmark item embedded inside a much longer training record (for example, one question inside a long multi-question prompt) shares only a small fraction of that record’s shingles. It scores low and can be missed.
• Paraphrased, translated or reformatted copies are not detected unless they keep enough of the same wording. This method compares surface text, not meaning.
• Only the prompt side was compared. Answers, solutions, MMLU’s answer choices and HumanEval’s reference solutions were not. An item whose answer leaked without its question is not counted.
• A match on a short, general question is weak evidence of copying. For MMLU test items, the compared text is the question field alone, without the answer choices. A one-line general-knowledge question can appear word for word in a training set that was never built from the benchmark. The word counts next to each example show how long the matched texts are.
• A match is text overlap in a dataset, not evidence about a model. Nothing here says whether any model trained on these datasets, or how any model’s benchmark score would change.
• Results are for the pinned versions only. A dataset updated after the pinned commit may differ.

Why there is no benchmark text here

Publishing test items on the open web is one way they end up in the next training crawl. So this page deliberately does not quote any benchmark question, prompt or solution, and neither does the data file it is built from. Every match is given as a position in a pinned file (and a subject or task id where the benchmark has one), which is enough to find it without reprinting it. The only thing recorded about the matched text itself is its length in words.

Check your own data

• Check your own train and eval split in the free browser checker. Same three levels, same code, with the text fields to compare picked automatically. Your files stay on your machine.
• Run it in CI with the command-line toolkit, so a pull request that adds overlap fails before it merges.
• Want a training set and benchmark pair measured? Tell us.