Providers

Providers

The judge runs on any of these backends. Pick one with JUDGE_PROVIDER or the --provider flag. Nothing here requires a paid service.

Provider	API key	Runs	Notes
groq (default)	yes (free)	cloud	Free plan, rate-limited per model. Default judge openai/gpt-oss-120b.
ollama	no	your machine	Free local models via Ollama: ollama pull llama3.2. Verified live weekly in CI with qwen2.5:1.5b.
openai_compatible	optional	either	Any /chat/completions server: LM Studio, vLLM, llama.cpp, or hosted free models. Verified live against Groq's OpenAI-compatible endpoint (https://api.groq.com/openai/v1).
mock	no	offline	Deterministic heuristic, not an LLM. For demos, tests and trying the tool without a key.

llm-judge providers                      # what's available and configured
llm-judge --provider ollama health       # reachability check, no inference
llm-judge --provider mock evaluate --task "..." --output-a "..." --output-b "..."

from llm_judge import Judge
from llm_judge.providers import OllamaProvider, OpenAICompatibleProvider

judge = Judge(provider=OllamaProvider("qwen2.5:7b"))
judge = Judge(provider=OpenAICompatibleProvider("http://localhost:1234/v1", "local-model"))

Every result records which provider and model produced it (metadata.judge), and the library never switches provider or model silently. Mock results say so in every explanation.

Fallback (opt-in). Set JUDGE_FALLBACK_PROVIDERS (for example ollama) to try other providers, in order, when a call to the primary one fails. It is off by default and never silent: - a result answered by a fallback carries a provider_fallback warning; - it also carries metadata.fallback entries naming the provider that failed, the one that answered, and the error; - metadata.judge keeps naming the configured judge.

Two things never trigger a fallback: an exhausted demo budget, and a reply that fails validation (the repair retry handles that). The mock judge can't be a fallback, because its scores aren't LLM judgments. Third-party packages can add providers through the llm_judge.providers entry-point group (see llm_judge/providers/registry.py).

Small local models are weaker judges than large hosted ones. Expect noisier scores and more repair retries, and prefer a judge from a different model family than the candidates (the self_preference_risk warning flags this).