Documentation

Install

The toolkit ships as a zip containing a standard Python package. Python 3.9 or newer; the core has no required dependencies.

unzip splitcheck-toolkit-v1.0.0.zip
cd toolkit
pip install .

# or, isolated:
pipx install ./toolkit
uv tool install ./toolkit

Optional extras:

pip install './toolkit[parquet]'   # read .parquet (pyarrow)
pip install './toolkit[hf]'        # Hugging Face datasets helpers

Check it worked:

splitcheck --version

Quick start

splitcheck check train.jsonl eval.jsonl

SplitCheck
  train: train.jsonl (48,120 records, jsonl)
  eval:  eval.jsonl (1,000 records, jsonl)
  fields: text
  levels: exact, normalized, near   threshold: 0.8

  exact:      31 eval record(s), 31 train record(s), 31 pair(s)
  normalized: 12 eval record(s), 12 train record(s), 12 pair(s)
  near:       27 eval record(s), 34 train record(s), 41 pair(s)

70 of 1,000 eval records (7.00%) also appear in the train file.

CLI reference

splitcheck check TRAIN EVAL
    [--field text]
    [--fields prompt,completion]
    [--threshold 0.8]
    [--levels exact,normalized,near]
    [--report report.md]
    [--json report.json]
    [--clean train.clean.jsonl]
    [--fail-over 0.0]
    [--train-format jsonl|csv|tsv|txt|parquet]
    [--eval-format  jsonl|csv|tsv|txt|parquet]
    [--permutations 128]
    [--max-pairs 5000]
    [--quiet]

Choosing fields

With neither --field nor --fields, SplitCheck auto-detects. It looks for a known pair first — prompt+completion, instruction+output, question+answer, input+output, prompt+response — then for a single known column:

text · prompt · input · question · instruction · completion · output · answer · response

When several fields are selected they are joined with a newline and compared as one string. That is usually what you want for instruction data: an identical prompt with a different completion is not the same example.

Exit codes

2 always means “SplitCheck could not do its job”, never “your data is dirty”. That distinction is what lets you tell a real finding from a broken pipeline.

Using it in CI

Fail the build on any overlap:

splitcheck check data/train.jsonl data/eval.jsonl --report contamination.md

Allow up to 0.5% — some benchmarks legitimately share boilerplate:

splitcheck check data/train.jsonl data/eval.jsonl --fail-over 0.005

Report and clean without ever failing the job:

splitcheck check data/train.jsonl data/eval.jsonl \
  --clean data/train.clean.jsonl \
  --json contamination.json \
  --fail-over 1.0

GitHub Action

The zip includes this at toolkit/.github/workflows/splitcheck.yml.

name: Contamination check
on: [pull_request]

jobs:
  splitcheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install SplitCheck
        run: pip install ./toolkit
      - name: Check train/eval overlap
        run: |
          splitcheck check data/train.jsonl data/eval.jsonl \
            --report contamination.md \
            --json contamination.json \
            --fail-over 0.0
      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: contamination-report
          path: |
            contamination.md
            contamination.json

if: always() matters — without it the report is not uploaded on the run where you most want to read it.

Python API

from splitcheck import check_files

result = check_files(
    "data/train.jsonl",
    "data/eval.jsonl",
    fields=["prompt", "completion"],
    threshold=0.85,
    levels=["exact", "normalized", "near"],
)

report = result.report
report.contamination_rate        # 0.07
report.counts["near"].pairs      # 41
report.pairs[0].eval_snippet

# Indices into the original train list, in original order.
result.kept_train_indices
result.removed_train_indices

Working from records you already hold in memory:

from splitcheck import check_records
from splitcheck.detect import DetectOptions

result = check_records(
    train_records,   # list[dict]
    eval_records,    # list[dict]
    DetectOptions(fields=["text"], threshold=0.8),
)

Rendering the reports yourself:

from splitcheck.report import report_to_markdown, report_to_json, verdict

open("report.md", "w").write(report_to_markdown(result.report))
open("report.json", "w").write(report_to_json(result.report))
print(verdict(result.report))

With the hf extra, Hugging Face splits convert straight to records:

from datasets import load_dataset

train = load_dataset("my/dataset", split="train").to_list()
evalset = load_dataset("my/dataset", split="test").to_list()
result = check_records(train, evalset, DetectOptions(fields=["text"]))

Formats

• JSONL — one JSON object per line. A top-level JSON array is also accepted.
• CSV / TSV — a header row is required. Quoted fields, escaped quotes and embedded newlines are handled.
• TXT — one record per line, exposed as the text field.
• Parquet — with the parquet extra installed.

Format is inferred from the extension; override it with --train-format / --eval-format.

How detection works

Three levels, strongest first. A pair is counted once, at the strongest level it matched — an exact duplicate is never also counted as a near-duplicate.

1. Exact — the selected fields are joined and compared byte for byte. Complete.
2. Normalized — Unicode NFKC, lowercased, punctuation and symbols stripped, whitespace collapsed. Also complete.
3. Near-duplicate — word 5-gram shingles (character 8-grams for texts shorter than five words), 128-permutation MinHash signatures, LSH banding to generate candidates, then Jaccard on the shingle sets to score them. Records with more than 512 distinct shingles are reduced to a bottom-k sketch (the 512 smallest hashes), which keeps long records cheap without letting a small edit shift the sample.

The approximation is in candidate generation, not in the MinHash signatures: scores never come from signature agreement. LSH can miss a small number of borderline pairs near the threshold, so near-duplicate recall is high but not guaranteed to be 100%. Similarities are exact for records up to 512 distinct shingles, and an unbiased bottom-k estimate above that.

The toolkit and the free browser checker use the same hash constants and the same banding plan, so they agree on the same input.

Limits

• Records longer than 512 shingles are sampled down with a deterministic stride, so one enormous document cannot dominate a run.
• --clean removes every train record that matched. That is the safe default. To review first, export --json and filter yourself.
• Semantic paraphrases that share no 5-gram are not found. Nothing shingle-based will find them; that needs embeddings, which is a slower and less deterministic tool.
• Comparison is over the text you select. Contamination hiding in a field you did not select is invisible.

Support

Get in touch with your license key, the SplitCheck version, and — if you can — a small file that reproduces the problem.

Not bought it yet? The toolkit is $29, one-time. The browser checker on the home page stays free.