Getting started¶

Installation¶

Python 3.11 or later is required. The package is on PyPI as llm-as-judge-platform; it installs the llm_judge library and the llm-judge command:

pip install llm-as-judge-platform              # library and CLI
pip install "llm-as-judge-platform[all]"       # plus dashboard, REST API, YAML, PostgreSQL

To run the dashboard or work on the code, clone the repository:

git clone https://github.com/syed-waleed-ahmed/LLM-as-Judge.git
cd LLM-as-Judge

python -m venv .venv
# Windows:  .venv\Scripts\activate
# Unix:     source .venv/bin/activate

# Library + UI + dev tooling
pip install -e ".[ui,dev]"

# Or everything (UI, REST API, YAML, PostgreSQL)
pip install -e ".[all]"

Every GitHub Release also has the wheel and sdist attached. The dashboard lives in app.py and dashboard/, so it runs from a clone.

Upgrading an environment from before 2.3.0? The distribution was renamed from llm-as-judge, so remove the old one first with pip uninstall llm-as-judge. Otherwise its files can shadow the new install.

Then configure your key:

cp .env.example .env      # then edit .env and set GROQ_API_KEY

Get a free key at https://console.groq.com/keys. No key yet? Every command works offline with --provider mock, a deterministic heuristic judge (clearly labelled, not an LLM), or locally with --provider ollama.

Usage¶

1. Web app (Streamlit)¶

streamlit run app.py

A multi-page dashboard over the same engine:

Page	What it does
Overview	Which judge is active, session and stored-run statistics, recent runs
Playground	Compare two outputs, grade one, check against a reference, or rank candidates, with optional judge instructions, charts, explanations and warnings
Datasets	Upload CSV/JSON/JSONL/YAML (or pick a sample), validate, evaluate, chart and download results
Benchmarks	Run a bundled sample and see per-system statistics
RAG evaluation	Faithfulness, answer relevance, context precision and recall, with claim-level verdicts
Agent evaluation	Paste or upload a trace: deterministic checks plus a judged grade
Experiments	Compare two recorded runs: verdict, confidence interval, per-row changes; download Markdown reports
Analytics	Across recorded runs: score trends, score distributions, per-criterion comparison, bias signals and operational reliability
Reliability	Repeated-judgment consistency and agreement between judges
Settings	Judge temperature and tie margin for your session, provider status and health checks, limits in effect, storage, cache and privacy controls
About	How the results stay honest, privacy, how to run it yourself

The sidebar switches between the live judge (when a key is configured) and the offline mock judge. The mock judge works without any key, and its results are always labelled as not coming from an LLM. Charts use a colorblind-validated palette with separate light and dark versions.

For a public deployment, set JUDGE_DEMO_MODE=true. That adds per-session and daily live-call budgets and input caps, and keeps each visitor's data session-only. See DEPLOYMENT.md for free Streamlit Community Cloud hosting.

2. Command line¶

# List built-in criteria (no API key required)
llm-judge list-criteria

# Judge a single pair
llm-judge evaluate \
  --task "Write a friendly tweet announcing our new AI tool" \
  --output-a "Meet our new AI assistant - it saves you hours every week. Try it free!" \
  --output-b "We have released a new artificial intelligence product." \
  --criteria "Creativity,Brand Tone Adherence" \
  --mitigate-bias

# Weight criteria and record which models produced each output
llm-judge evaluate --task-file task.txt \
  --output-a-file a.txt --output-b-file b.txt \
  --criteria "Helpfulness,Conciseness" --weights "Helpfulness=2" \
  --model-a gpt-4o --model-b llama-3.1-8b-instant

# Read long inputs from files and emit raw JSON
llm-judge evaluate --task-file task.txt \
  --output-a-file a.txt --output-b-file b.txt --json

# Give the judge your own guidance (also: grade, rank, batch, benchmark)
llm-judge evaluate --task-file task.txt \
  --output-a-file a.txt --output-b-file b.txt \
  --instructions "Penalise answers that state facts without a source."

# Grade a whole dataset concurrently (checkpointed; resumable with --resume)
llm-judge batch --input examples/sample_batch.csv \
  --output results.csv --criteria Helpfulness --workers 8

# Run a bundled benchmark sample and print per-model statistics
llm-judge benchmark summarization

# Grade one output on its own, or against an expected answer
llm-judge grade --task "Explain DNS" --output-file answer.txt
llm-judge grade --task "When did Apollo 11 land?" \
  --output "In 1969." --reference "Apollo 11 landed on the Moon on 20 July 1969."

# Rank several candidates (round robin, or --strategy pointwise for fewer calls)
llm-judge rank --task "Write a tagline" --candidates-file candidates.json

candidates.json is either {"id": "text", ...} or a list of {"id": ..., "output": ..., "model": ...} objects.

3. Python library¶

from llm_judge import Criterion, Judge, resolve_criterion

judge = Judge()  # reads config from environment / .env

result = judge.evaluate(
    task_description="Summarise the text in one sentence.",
    output_a="A concise, accurate one-line summary.",
    output_b="A rambling three-paragraph response.",
    criteria=["Helpfulness", "Conciseness"],
    candidate_a_model="gpt-4o",  # optional; recorded in metadata
)

print(result.overall_winner)  # Winner.A
print(result.overall_score_a)  # weighted, scale-normalised score (0-1)
for warning in result.warnings:  # reliability signals worth a second look
    print(warning.code, warning.message)

# Custom criteria can set their own scale and weight.
coverage = Criterion(
    name="Coverage", description="Covers every key point.", scale_min=0, scale_max=100, weight=2
)
judge.evaluate("...", "...", "...", [resolve_criterion("Helpfulness"), coverage])

# Optional guidance for the judge: how strictly to score, what to penalise.
# It's recorded in result.metadata["instructions"] and in experiment runs.
judge.evaluate(
    "...",
    "...",
    "...",
    "Accuracy",
    instructions="Penalise answers that state facts without a source.",
)

The same Judge also grades single outputs and ranks candidates:

# Pointwise: grade one output on its own merits.
grade = judge.evaluate_pointwise("Explain DNS.", answer, ["Helpfulness", "Conciseness"])
print(grade.overall_score, grade.feedback)

# Reference-based: compare with an expected answer (defaults to Correctness + Completeness).
check = judge.evaluate_reference("When did Apollo 11 land?", "In 1968.", reference="20 July 1969.")
print(check.unsupported_claims)  # hallucination indicators
print(check.missing_information)  # key points the output omits
print(check.reference_metrics)  # exact_match, token_f1, optional semantic_similarity

# Ranking: round robin (every pair head to head) or pointwise (cheaper).
ranking = judge.rank("Write a tagline", {"v1": "...", "v2": "...", "v3": "..."})
for row in ranking.leaderboard:
    print(row.rank, row.candidate, row.score, (row.wins, row.losses, row.ties))

Every method has an async twin (aevaluate, aevaluate_pointwise, aevaluate_reference, arank, aevaluate_rag, aevaluate_agent). Each one runs the evaluation in a worker thread, so you can await it from async code without blocking the event loop, and run several concurrently:

import asyncio

results = await asyncio.gather(
    judge.aevaluate(task, a, b, "Helpfulness"),
    judge.aevaluate_pointwise(task, a, "Conciseness"),
)

Set JUDGE_CACHE_SIZE (for example 256) to reuse results of identical evaluations: same inputs, criteria, instructions, judge and settings. The cache is opt-in, in memory only, and used only at temperature 0. Hits are marked metadata["cache_hit"], and they don't spend provider quota. Consistency analysis never uses it.

Every pairwise evaluation returns a validated, versioned result:

{
  "schema_version": "1.3",
  "criteria": [
    {
      "criterion": "Helpfulness",
      "output_a_score": 8.0,
      "output_b_score": 3.0,
      "output_a_explanation": "Directly and accurately answers the task.",
      "output_b_explanation": "Vague and largely off-topic.",
      "winner": "A"
    }
  ],
  "overall_winner": "A",
  "overall_comment": "Output A is clearly more helpful.",
  "overall_score_a": 0.7778,
  "overall_score_b": 0.2222,
  "warnings": [],
  "metadata": {
    "model": "openai/gpt-oss-120b",
    "judge": { "provider": "groq", "model": "openai/gpt-oss-120b" },
    "candidates": { "A": { "model": "gpt-4o" }, "B": { "model": null } },
    "prompt_version": "2.1",
    "position_bias_mitigated": false,
    "latency_seconds": 0.93
  }
}

4. REST API¶

pip install -e ".[api]"              # from a clone; included in ".[all]"
llm-judge serve                      # http://127.0.0.1:8000, docs at /docs

curl -s http://127.0.0.1:8000/api/v1/evaluate/pairwise \
  -H "Content-Type: application/json" \
  -d '{"task_description": "Explain DNS", "output_a": "...", "output_b": "...",
       "criteria": ["Helpfulness", {"name": "Accuracy", "weight": 2}]}'

Endpoint	Purpose
`POST /api/v1/evaluate/{pairwise,pointwise,reference,rank}`	Single evaluations
`POST /api/v1/evaluate/{rag,agent}`	RAG and agent-trajectory evaluation
`POST /api/v1/analysis/consistency`, `GET /api/v1/runs/agreement`	Judge reliability
`POST /api/v1/evaluate/batch`	Small synchronous batches (`JUDGE_API_MAX_BATCH_ROWS`)
`POST /api/v1/jobs`, `GET /api/v1/jobs/{id}[/rows]`, `DELETE /api/v1/jobs/{id}`	Background dataset jobs (bounded queue, cancellable, idempotent with `Idempotency-Key`)
`POST /api/v1/datasets/validate`	Validate rows or CSV/JSONL/JSON/YAML content
`GET /api/v1/benchmarks`, `POST /api/v1/benchmarks/{name}/jobs`	Bundled sample benchmarks
`GET /api/v1/experiments`, `/runs`, `/runs/{id}[/rows]`, `/runs/compare`	Experiment history and regression checks
`GET /api/v1/system/status`, `/providers`, `/criteria`, `/statistics`, `GET /health`	Service information

Protections, all configurable: - API keys. JUDGE_API_KEYS (comma-separated), sent as X-API-Key or Authorization: Bearer. - Admin routes. Deleting runs and listing all jobs are disabled unless JUDGE_API_ADMIN_KEY is set. - Rate limiting. A per-client limit, with Retry-After on 429. - Size limit. Request bodies over the limit get a 413. - Concurrency cap. Excess load gets a 503, so free-tier quotas aren't exhausted silently.

Every error uses one JSON envelope with a request id, and request logs are structured JSON without bodies. serve binds to localhost by default and warns when exposed without API keys.

No always-free host is assumed for the API. Run it locally or on your own infrastructure. The public demo (Streamlit) doesn't depend on it.