LLM-as-Judge

An open-source platform for evaluating LLM outputs with an LLM judge. One engine powers a Python library, a CLI, a REST API and a Streamlit dashboard, and everything runs on free tiers or entirely offline.

Get started Live demo API reference Source on GitHub

Why this exists

"LLM-as-Judge" is a common pattern for scoring subjective qualities — creativity, tone, code readability, factual accuracy — where there is no exact-match ground truth. Doing it well in production requires more than a single API call, and this project handles the parts that usually get skipped:

• Structured, validated output. The judge is pinned to a strict JSON schema (via the provider's JSON mode), and every response is validated with Pydantic. The returned criteria must match the requested ones exactly. A malformed reply gets a bounded "repair" retry that tells the judge what was wrong.
• Position-bias mitigation. LLM judges tend to favour whichever answer is shown first. Optionally, each pair is judged in both orders (concurrently) and the scores are averaged, cancelling that bias. The explanations shown always agree with the averaged verdict.
• Self-consistency. Out-of-range scores are clamped (and flagged), and winners are recomputed from the numbers, so the verdict never contradicts the scores. Criteria on different scales are normalised before they're combined.
• Injection resistance. Candidate outputs are treated as untrusted data, wrapped in boundary markers that carry a random per-request token.
• Bias diagnostics. Results carry warnings for suspected prompt injection, verbosity bias, self-preference risk and order-dependent verdicts.
• Resilience. Transient provider failures (rate limits, timeouts, 5xx) are retried with exponential backoff; permanent errors fail fast.
• Scale. A concurrent batch runner grades thousands of rows from CSV/JSONL, isolating per-row failures so one bad row never aborts the run.

Features

Capability	Description
Library, CLI, UI & REST API	One evaluation core, four ways to use it.
Evaluation modes	Pairwise (A vs B), pointwise, reference-based, multi-candidate ranking, RAG (faithfulness, context precision/recall) and agent trajectories.
Judge reliability	Repeated-judgment consistency with confidence intervals; inter-judge agreement (Cohen's kappa, Spearman).
Multi-criteria	Score on several criteria at once (built-in or custom), with optional weights.
Bias mitigation	Optional A/B order swapping to reduce position bias.
Reliability warnings	Flags prompt injection, verbosity bias, self-preference and order sensitivity.
Retries & repair	Backoff for transient errors; repair retries for malformed replies.
Experiment tracking	Run history in SQLite (default) or PostgreSQL/Supabase, run comparison with confidence intervals, Markdown reports, and a regression gate for CI.
Datasets & batches	CSV/JSON/JSONL/YAML validation, checkpointed and resumable batch runs, per-model summaries, bundled benchmark samples.
Fully tested	Offline unit and property-based tests (Hypothesis) with no network calls, plus CI jobs against a real PostgreSQL server and live Ollama models.
Pluggable providers	Groq, Ollama (local), any OpenAI-compatible API, and an offline mock; add your own by implementing one method.
Typed & linted	Pydantic models, mypy-clean, ruff-clean and ruff format-clean.

Screenshots

The dashboard with the live Groq judge. Try it in the live demo.

Technology stack

Area	Tools
Language and models	Python 3.11 to 3.14, fully typed (mypy); Pydantic v2 and pydantic-settings
LLM providers	Groq SDK; httpx for Ollama and OpenAI-compatible servers; an offline mock judge
Resilience	tenacity retries that follow provider rate-limit hints; opt-in fallback and result cache
REST API	FastAPI and Uvicorn
Dashboard	Streamlit and Plotly
Storage and data	SQLite (standard library) or PostgreSQL/Supabase (psycopg); CSV, JSON, JSONL and YAML (PyYAML)
Quality	pytest, Hypothesis, ruff, mypy, Bandit, pip-audit, CodeQL, pre-commit
Delivery	GitHub Actions, Docker, MkDocs Material on GitHub Pages, Streamlit Community Cloud

Every runtime dependency is open source, and every hosted service has a free tier.

Limitations

• LLM judges are imperfect. Position, verbosity and self-preference bias are mitigated and flagged, not eliminated. Scores are judgments, not ground truth. The bias warnings are heuristics.
• Free tiers have limits. Groq's free plan caps tokens per minute and per day. Batches slow down when they hit the cap (retries follow Groq's own wait hints). The public demo caps live calls, and its app sleeps when idle.
• The mock judge isn't an LLM. It's a text heuristic for demos and tests, and it ignores judge instructions. Its results are always labelled.
• Semantic similarity is optional. Reference grading always combines the judge's correctness and completeness scores with exact match and token F1, which are lexical. Embedding-based semantic similarity needs an embedding backend (JUDGE_EMBEDDINGS_PROVIDER). Both embedding backends are verified live in CI against Ollama; Groq offers no embeddings API.
• Single-node jobs. REST API jobs run in an in-process queue that doesn't survive a restart, and nothing scales horizontally. Recorded runs can live in PostgreSQL or Supabase, so they can outlive a single machine.
• The REST API isn't hosted publicly. Run it locally or on your own server.