Skip to content

Datasets, benchmarks and experiments

Datasets, batches and benchmarks

llm-judge batch evaluates whole datasets in CSV, JSON, JSONL or YAML (YAML needs the yaml extra, which all includes). The dataset kind is inferred from the columns:

Kind Required columns Optional
pairwise task_description, output_a, output_b id, model_a, model_b
pointwise task_description, output id, model
reference task_description, output, reference id, model

Common column names such as prompt, question, response and ground_truth are mapped automatically. An id column is optional; the row index is used if it's missing.

id,task_description,output_a,output_b,model_a,model_b
1,Write a tagline,"Fast. Simple. Yours.","Our product is good.",system-alpha,system-beta

llm-judge datasets validate data.csv     # kinds, duplicates, missing values, fingerprint
llm-judge batch --input data.csv --output results.csv --workers 8
llm-judge batch --input data.csv --output results.csv --resume   # after a crash or Ctrl+C
llm-judge batch --input big.jsonl --output sample.csv --sample 200 --seed 1

Batch runs are built for large, flaky workloads:

  • Checkpointing. Every finished row is appended to <output>.checkpoint.jsonl straight away. The checkpoint is deleted after a fully successful run and kept otherwise.
  • Resume. --resume skips rows that already succeeded and retries the failed ones. It refuses to continue if a row changed since it was checkpointed, so results from two dataset versions are never mixed.
  • Bounded resources. At most 2 x workers requests are in flight, and files over JUDGE_MAX_DATASET_BYTES or JUDGE_MAX_DATASET_ROWS are rejected before parsing.
  • Failure isolation. A failing row is recorded with its error and never aborts the run.

Each run ends with a summary: winners and per-model win rates for pairwise data, and mean scores per model, mean token F1 and rows with unsupported claims for graded data.

Benchmarks

Nine small, original, synthetic datasets ship with the package for demos and smoke tests: general_qa, code_generation, summarization, rag, instruction_following, safety, math_reasoning, customer_support and data_extraction. Their candidate "systems" are neutral labels, not real models.

llm-judge datasets samples                       # list them
llm-judge benchmark safety                       # run one with the configured judge
llm-judge --provider mock benchmark general_qa   # fully offline and reproducible
llm-judge benchmark my_dataset.jsonl --output rows.csv

These samples are tiny, so treat their numbers as a demonstration, not as evidence about real models.

Experiment tracking and regression detection

Add --experiment NAME to batch or benchmark to record the run in a local SQLite database (JUDGE_DB_PATH, default .llm_judge/experiments.db). That needs no server and no account.

To share runs across machines, point JUDGE_DB_PATH at a PostgreSQL URL instead. A free Supabase project works: install the extra (pip install -e ".[postgres]"), then set JUDGE_DB_PATH=postgresql://postgres.<ref>:<password>@<host>:6543/postgres, the "Transaction pooler" connection string from the project's Connect dialog. Tables are created on first use, and both backends store the same data. DEPLOYMENT.md covers choosing a pooler and keeping the tables out of Supabase's REST API.

Each run stores its per-row results with reproducibility metadata: dataset fingerprint, judge provider and model, prompt version, criteria, relevant settings, and the library and Python versions. Raw task text, outputs and references are not stored, only their hashes. Judge explanations may still quote them.

llm-judge batch --input qa_v1.jsonl --output v1.csv --experiment qa --run-name v1
llm-judge batch --input qa_v2.jsonl --output v2.csv --experiment qa --run-name v2
llm-judge experiments runs --experiment qa
llm-judge experiments compare <v1-run-id> <v2-run-id> --fail-on-regression

# A shareable Markdown report: how the run was judged, results, failures,
# reproducibility details, and (with --baseline) the comparison
llm-judge experiments report <v2-run-id> --baseline <v1-run-id> --output report.md

  mean: 0.574 -> 0.259 (change -0.315, 95% CI [-0.556, -0.093], bootstrap)
  verdict: REGRESSION
  rows regressed: 3; rows improved: 0
    worse  qa-1: 0.667 -> 0.000

Rows are paired by id. The comparison reports the mean paired difference with a seeded bootstrap 95% confidence interval (a normal approximation above 5,000 pairs), and gives a verdict of regression, improvement, no significant change or insufficient data. It also lists the rows that moved most. --fail-on-regression makes it usable as a CI gate. The runs are flagged as not directly comparable when they used a different judge, prompt version, criteria or bias setting.

These are judge-scored differences: the judge's own noise and biases are part of what is measured.