SplitCheck
Get the toolkit
Free · runs in your browser · no sign-in

Your eval set is probably in your training set.

SplitCheck finds the overlap: byte-identical records, records that only differ in casing and punctuation, and near-duplicates above a similarity threshold you choose. You get a report you can review and a training file with the contamination removed.

the file you train on
the file you benchmark on

Runs entirely in this browser tab. Your files are never uploaded.

How it works

Three passes over your data, strongest evidence first. A matched pair is counted once, at the strongest level it matched — so an exact duplicate never inflates the near-duplicate number too.

  1. 01

    Exact

    The selected fields are joined and compared byte for byte. A hash lookup, so it is complete: no misses, no false positives.

  2. 02

    Normalized

    Unicode NFKC, lowercased, punctuation and symbols stripped, whitespace collapsed. Catches the same example after a reformat, a re-quote, or a markdown pass. Also complete.

  3. 03

    Near-duplicate

    Each text becomes a set of word 5-gram shingles (character 8-grams when it is shorter than five words). MinHash signatures over 128 permutations are bucketed with LSH banding to find candidates cheaply, then every candidate is re-scored on the shingle sets themselves — exactly for records up to 512 distinct shingles, and with an unbiased bottom-k estimate over the 512 smallest hashes for longer ones.

Where the approximation is

LSH is a filter on which pairs get compared, not on how they are scored. It can miss a small number of borderline pairs near the threshold, so near-duplicate recall is high but not guaranteed to be 100%. Scoring always uses the shingle sets, never the MinHash signatures: it is exact Jaccard for records with up to 512 distinct shingles (roughly 500 words), and an unbiased bottom-k estimate over the 512 smallest hashes for longer ones. Exact and normalized matching have no approximation at all.

Contamination rate is eval records with ≥1 match ÷ total eval records. It counts records, not pairs, so one training file that duplicates a single eval example forty times still contributes one contaminated eval record.

SplitCheck Toolkit · $29 one-time

The same checks, offline and in CI

A Python package and CLI with the same three detection levels and the same hash constants, so it agrees with the browser checker on the same input. Run it on data you cannot put in a browser, wire it into a pull-request workflow, and fail the build when contamination goes above a rate you set.

Get the toolkit — $29Read the docs
$ splitcheck check train.jsonl eval.jsonl \
    --fields prompt,completion \
    --threshold 0.85 \
    --clean train.clean.jsonl \
    --report contamination.md \
    --fail-over 0.005

  exact:      31 eval, 31 train, 31 pairs
  normalized: 12 eval, 12 train, 12 pairs
  near:       27 eval, 34 train, 41 pairs

70 of 1,000 eval records (7.00%) also
appear in the train file.

$ echo $?
1

Questions

Do my files get uploaded?
No. The checker is a Web Worker running in your tab. Both files are read with the browser’s File API, compared locally, and the report is generated locally. There is no upload endpoint — open your network tab and run a check: no request carries your data. What this site does record is the page request itself (path and referrer, counted on our server, no cookies and no IP address stored) and, if the analytics script loads, an anonymous page view.
How big a file can it handle?
Roughly 50k × 50k records finishes in about half a minute on a modern laptop, and that is the size we tuned for. Beyond that you are limited by browser memory rather than by the algorithm — a browser tab typically gives you 2–4 GB. If your datasets are bigger than that, or you want this in CI, the toolkit runs the same three levels offline with no size ceiling.
Is this a new algorithm?
No, and we would rather say so plainly. Exact and normalized matching are hash lookups. Near-duplicate detection is MinHash with LSH banding — a textbook technique from 1997 that ships in several open-source libraries. Free alternatives exist. What you are getting here is packaging: no install for the free tier, a report you can hand to a colleague, a cleaned file in one click, and a CLI that returns a useful exit code.
Will it find paraphrases?
Only lexical ones. If two records share word 5-grams — reworded intros, changed numbers, added or removed sentences — they will be found. A genuine paraphrase that reuses no 5-gram will not be. Catching those needs embeddings, which is slower, non-deterministic and a different tool. We would rather be exact about what this does than imply it reads meaning.
Which formats can I use?
JSONL (and a top-level JSON array), CSV and TSV with a header row, and plain text with one record per line. The toolkit adds Parquet through an optional extra. Fields are auto-detected — text, prompt+completion, question+answer and friends — and you can override the choice before running.
What exactly does the cleaned file contain?
Your original training records, byte for byte, minus every record that matched an eval record at the levels you selected — written as JSONL. Nothing is rewritten or re-serialised beyond the JSON encoding. If you would rather review before deleting, export report.json: it lists every matched pair with both indices.

Want the longer version? Read the guide to train/test contamination.