Delivery report: LLM-as-Judge 2.4

This report closes the platform plan that started with the Phase 0 audit. Version 2.1 completed the plan, and 2.2 took on the "What's next" items. The report covers what was built, how it was verified, and what is still open. It follows the plan's final-report checklist. Three labels are used throughout:

• Implemented and tested: covered by the offline test suite and CI.
• Implemented, not externally verified: tested only against simulated services, because the real service wasn't available for free.
• Planned, not implemented: deliberately deferred, with the reason.

1. Final architecture

One evaluation core, four interfaces:

• Interfaces: the Python library, the llm-judge CLI, a FastAPI REST API and a Streamlit dashboard.
• Application layer: datasets with resumable batches, experiment tracking in SQLite, comparison and reports, reliability analytics, and a bounded in-process job queue.
• Providers: Groq, Ollama, OpenAI-compatible and an offline mock judge, behind one protocol. Optional layers wrap them: demo quotas, an opt-in fallback chain and an opt-in result cache.

The core library has no UI dependencies. Diagrams, the request flow and the design principles are in ARCHITECTURE.md.

2. Implementation summary

Release	What it delivered
2.0.0	Phases 0 to 11 of the plan: hardened core, new evaluation modes, providers, datasets, experiments, REST API, dashboard, RAG and agent evaluation, security hardening, public demo, docs site and release automation. Two independent code reviews and browser end-to-end tests found 32 defects, all fixed.
2.0.1	Open-source readiness (community files, CodeQL, Dependabot), docs-site link fixes, and a Streamlit 1.67 dashboard fix found in browser testing.
2.1.0	The gaps a line-by-line review against the plan found: judge instructions, provider fallback, async methods, a result cache, evaluation reports, the Analytics and Settings pages, semantic similarity, a free-tier rate-limit fix, screenshots, a coverage badge and Python 3.10 deprecation.
2.2.0	The next phases: a PostgreSQL/Supabase experiment store tested in CI against a real server, live CI checks of the Ollama provider and embeddings, and three more benchmark samples.
2.3.0	Published on PyPI as llm-as-judge-platform, with one-click releases from the Actions tab. Python 3.10 support ended. OpenAI-compatible chat and embeddings are verified live in CI.
2.4.0	Resolved all CodeQL alerts (request data can't reach a file path; no hashing of the API key), a PyPI page with working screenshots and links, and the PostgreSQL store verified against a live Supabase project.

Since v1.0.0, 154 files changed (+22,380 / -610 lines, excluding generated graph data and screenshots).

3. Features implemented

Implemented and tested

• Evaluation:
• pairwise (multi-criteria, weights, scales, tie detection, position-bias mitigation);
• pointwise grading;
• reference grading (correctness, completeness, missing information, unsupported claims, exact match, token F1);
• ranking (round robin and pointwise leaderboards);
• custom criteria and judge instructions.
• Advanced:
• RAG evaluation: faithfulness, answer relevance, context precision and recall, claim-level verdicts;
• agent trajectory checks and grading;
• consistency analysis with confidence intervals;
• inter-judge agreement (Cohen's kappa, Spearman).
• Reliability safeguards:
• injection-resistant fenced prompts;
• strict schema validation with repair retries;
• exact criteria matching;
• verbosity, self-preference, position and injection warnings;
• provider rate-limit hints honoured across workers.
• Providers: Groq (live), OpenAI-compatible (live), Ollama (live in CI), the mock judge, a plugin registry, health checks, an opt-in fallback chain (live) and demo quotas.
• Datasets: CSV, JSON, JSONL and YAML with validation, duplicates, missing values, fingerprints, splitting and limits, plus checkpointed, resumable batches and nine synthetic benchmark samples.
• Experiments: a SQLite or PostgreSQL/Supabase store with reproducibility metadata, run comparison with bootstrap confidence intervals, a CI regression gate, Markdown reports, and experiment and run routes.
• Interfaces:
• the Python library, with typed async twins of every evaluation method;
• CLI commands: evaluate, grade, rank, rag, agent, batch, benchmark, datasets, experiments (including report), consistency, providers, health, serve and version;
• the REST API: versioned routes, background jobs, API keys, rate limits, size limits and load shedding;
• the dashboard: 11 pages, including Analytics and Settings.
• Engineering:
• an opt-in result cache;
• a security threat model with tested mitigations;
• a Docker image;
• an MkDocs site with a link-check build hook;
• release notes generated from the changelog.

Verified live in CI, since 2.2. The "Live Ollama" workflow runs these weekly against a real Ollama server: - the Ollama provider (qwen2.5:1.5b); - Ollama embeddings for semantic similarity (nomic-embed-text).

The PostgreSQL store runs its tests against a real PostgreSQL 16 server in CI. On 2026-11-11 the same tests also passed against a live Supabase project on the free plan, through both the transaction pooler and the session pooler, and a dataset run recorded through the CLI read back from it. This was a one-off manual check: CI has no Supabase credentials.

Since 2.3, the same workflow also verifies the openai_compatible provider and OpenAI-compatible embeddings, through Ollama's OpenAI-compatible /v1 API.

4. Features intentionally deferred

Planned, not implemented

• Public hosting for the REST API. No verified always-free host exists, so it's documented for local or self-hosted use, as the plan allows.
• Distributed job execution and horizontal scaling. The job queue is in-process behind a replaceable interface, and no scaling is claimed.
• The plan's suggested folder layout. It was given as "approximately", and the existing src layout was kept to avoid an unnecessary rewrite.

5. Changed files

Area	Files changed since v1.0.0
src/llm_judge/ (library, providers, datasets, experiments, API)	53
dashboard/ and app.py	16
tests/	32
docs/ (site pages, design notes, screenshots)	20
scripts/ (E2E, release notes, coverage badge, link hook)	5
.github/ (workflows, issue forms, templates)	10
Project files (README, ARCHITECTURE, SECURITY, ROADMAP, CHANGELOG and more)	the rest

The changelog lists every change, release by release.

6. New dependencies

• Runtime: httpx, for Ollama, OpenAI-compatible and embedding backends. The v1 dependencies groq, pydantic, pydantic-settings, tenacity and python-dotenv remain.
• Optional extras:
• api: fastapi, uvicorn;
• ui: plotly, pandas (with streamlit);
• yaml: pyyaml;
• postgres: psycopg (PostgreSQL / Supabase store);
• docs: mkdocs, mkdocs-material.
• Development: hypothesis, bandit, pre-commit, types-PyYAML, plus newer ruff and mypy.

All are open source, and none needs a paid service.

7. Test results

621 offline tests pass, with no API key or network access. They include unit and property-based tests (Hypothesis), and CLI, REST API, Streamlit AppTest, concurrency, docs-sync and security tests. CI runs them on Python 3.11, 3.12, 3.13 and 3.14.

Live and browser checks, run during development:

• Browser end-to-end: 53 of 53 checks in a real browser, with live Groq judging, on Streamlit 1.58 and 1.67 (scripts/e2e_dashboard.py).
• Deployed public demo: live judging, quota countdown and every page checked in a browser.
• Live Groq: every CLI mode, the REST API, judge instructions, the fallback chain (Ollama down, Groq answering) and a rate-limit stress test (11 of 11 rows).
• OpenAI-compatible provider: live against Groq's OpenAI-compatible endpoint.
• Ollama (CI): provider health, pairwise and pointwise judgments, embedding similarity ordering, and reference grading with semantic similarity, all passing against the real server.
• PostgreSQL (CI and locally): the store tests pass against PostgreSQL 16 (16.2 locally).

8. Coverage results

About 98% line coverage of the llm_judge package, with the PostgreSQL store included. The README badge is measured on every deploy from main, so it always shows the current figure.

9. Security audit results

• Static analysis: Bandit is clean (three suppressions, reviewed and documented). CodeQL scans Python and the workflows and has no open alerts.
• Dependencies:
• pip-audit is clean on Python 3.12 with the dashboard extra.
• Python 3.10 support ended in 2.3.0, because the fixed releases of Pillow and others need 3.11+.
• Repository: secret scanning and push protection are on, as are private vulnerability reporting, Dependabot alerts and security updates, and a protected main (pull requests with passing checks).
• Threat model: SECURITY.md covers each risk with a tested mitigation: prompt injection, hostile datasets (including YAML alias bombs), CSV injection, secret leakage, quota abuse and API abuse.

10. Performance results

From BENCHMARKS.md: the library adds about 0.23 ms per judgment, and throughput scales almost linearly with workers until the provider becomes the limit. A live Groq judgment takes about 1 second.

On Groq's free plan (8,000 tokens per minute), a batch now waits out rate limits as one: every worker pauses until the provider's own retry-after hint expires. Back-to-back benchmarks went from 8 of 11 to 11 of 11 rows.

11. GitHub Actions status

Workflow	Runs on	Jobs
CI	every pull request and push to main	tests on 3.11 to 3.14, audit, the PostgreSQL store against a service container, Bandit, Docker build and smoke tests, strict docs build, package build and smoke test
CodeQL	pull requests, main, weekly	Python and Actions analysis
Live Ollama	weekly, on demand, and pull requests touching those backends	the Ollama provider and embeddings against the official ollama/ollama image
Docs	push to main	site build, coverage badge, GitHub Pages deploy
Release	Run workflow button, or version tags	build, tag, GitHub Release with changelog notes, PyPI
Dependabot	monthly	grouped pip and Actions updates

main only accepts changes whose CI and CodeQL checks pass.

12. Deployment status

Target	Status
Public demo	Live on Streamlit Community Cloud (Python 3.12, demo mode, live Groq judging)
Documentation	Live on GitHub Pages, deployed from main
Releases	GitHub Releases with the wheel and sdist; release notes come from the changelog
PyPI	Published as llm-as-judge-platform
REST API	Local or self-hosted only (llm-judge serve, or the Docker image)

13. Public demo URL

https://llm-as-judge.streamlit.app/

14. GitHub repository URL

https://github.com/syed-waleed-ahmed/LLM-as-Judge

15. Exact setup instructions

git clone https://github.com/syed-waleed-ahmed/LLM-as-Judge.git
cd LLM-as-Judge
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[all]"
cp .env.example .env        # set GROQ_API_KEY (free), or use --provider mock

llm-judge --provider mock benchmark general_qa      # works with no key
streamlit run app.py                                # dashboard
llm-judge serve                                     # REST API, docs at /docs

Deploying your own demo, the REST API, the docs site and releases is covered step by step in DEPLOYMENT.md.

16. Free-tier services and their limitations

Service	Used for	Limits that matter
Groq free plan	Live judging	Per-model limits on requests and tokens. openai/gpt-oss-120b allowed 8,000 tokens per minute during testing. Retries follow Groq's wait hints.
Streamlit Community Cloud	Public demo	Sleeps when idle (about 30 s to wake) and has limited resources. Demo quotas: 25 live calls per session and 500 per day.
GitHub Actions and Pages	CI, docs, releases	Free for public repositories
Ollama	Optional local judging and embeddings	Your own hardware. Small local models are noisier judges.

17. Known issues

• LLM judges are imperfect. Bias mitigation and warnings reduce position, verbosity and self-preference effects; they don't remove them.
• The mock judge is a heuristic for demos and tests. It ignores judge instructions and is always labelled.
• In-memory and in-process state. The result cache lives in memory, and REST API jobs don't survive a restart.

18. Future roadmap

Open items, which contributions are welcome on, are in ROADMAP.md: - more providers and benchmark samples.