Configuration
Configuration¶
All settings are read from the environment (or .env). See
.env.example.
| Variable | Default | Description |
|---|---|---|
JUDGE_PROVIDER |
groq |
groq, ollama, openai_compatible or mock. |
GROQ_API_KEY |
— | Required for the groq provider. |
GROQ_MODEL |
openai/gpt-oss-120b |
Groq-hosted model to use. |
JUDGE_OLLAMA_BASE_URL |
http://localhost:11434 |
Ollama server. |
JUDGE_OLLAMA_MODEL |
llama3.2 |
Local model to judge with. |
JUDGE_OPENAI_BASE_URL |
— | Base URL of an OpenAI-compatible API (e.g. .../v1). |
JUDGE_OPENAI_MODEL |
— | Model name on that endpoint. |
JUDGE_OPENAI_API_KEY |
— | Optional bearer token for that endpoint. |
JUDGE_OPENAI_JSON_MODE |
true |
Send response_format: json_object (disable for servers that reject it). |
JUDGE_FALLBACK_PROVIDERS |
(empty) | Opt-in fallback providers, comma-separated and tried in order when the primary fails (not mock). |
JUDGE_TEMPERATURE |
0.0 |
Sampling temperature (0 = deterministic). |
JUDGE_REQUEST_TIMEOUT |
60 |
Per-request timeout in seconds. |
JUDGE_MAX_RETRIES |
3 |
Retries for transient failures. |
JUDGE_MAX_PARSE_RETRIES |
1 |
Repair retries for malformed or schema-violating replies. |
JUDGE_MITIGATE_POSITION_BIAS |
false |
Judge each pair in both orders. |
JUDGE_PARALLEL_BIAS_RUNS |
true |
Run the two orderings concurrently. |
JUDGE_TIE_MARGIN |
0.06 |
Normalised (0-1) score gap treated as a tie (just over half a point on a 1-10 scale). |
JUDGE_VERBOSITY_RATIO |
2.0 |
Length ratio at which a long winner is flagged. |
JUDGE_MAX_CRITERIA |
20 |
Maximum criteria per evaluation. |
JUDGE_MAX_CANDIDATES |
8 |
Maximum candidates per ranking (round robin makes n(n-1)/2 comparisons). |
JUDGE_MAX_INPUT_CHARS |
50000 |
Longer inputs are truncated (with a warning). |
JUDGE_MAX_INSTRUCTIONS_CHARS |
2000 |
Longest judge guidance (instructions); longer text is truncated with a warning. |
JUDGE_CACHE_SIZE |
0 (off) |
Opt-in in-memory cache of identical temperature-0 results; hits are marked cache_hit. |
JUDGE_EMBEDDINGS_PROVIDER |
(empty) | ollama or openai_compatible: adds embedding-based semantic_similarity to reference grading. |
JUDGE_EMBEDDINGS_MODEL |
(empty) | Embedding model, e.g. nomic-embed-text on Ollama. |
JUDGE_EMBEDDINGS_BASE_URL / JUDGE_EMBEDDINGS_API_KEY |
(provider defaults) | Override the embeddings endpoint and key (default: the Ollama or OpenAI-compatible settings). |
JUDGE_API_KEYS |
— | Comma-separated keys required by /api/v1 (open if unset). |
JUDGE_API_ADMIN_KEY |
— | Enables admin routes (they're disabled if unset). |
JUDGE_API_RATE_LIMIT |
60 |
Requests per minute per client (0 disables). |
JUDGE_API_MAX_BODY_BYTES |
1000000 |
Larger request bodies get 413. |
JUDGE_API_MAX_CONCURRENT |
4 |
Concurrent evaluations before 503. |
JUDGE_API_MAX_BATCH_ROWS / JUDGE_API_MAX_JOB_ROWS |
50 / 1000 |
Row limits for sync batches and jobs. |
JUDGE_API_MAX_QUEUED_JOBS |
10 |
Waiting jobs before 429. |
JUDGE_API_CORS_ORIGINS |
— | Comma-separated allowed origins (CORS off if unset). |
JUDGE_DEMO_MODE |
false |
Public-demo safeguards in the dashboard (quotas, caps, session-only storage). |
JUDGE_DEMO_SESSION_CALLS / JUDGE_DEMO_DAILY_CALLS |
25 / 500 |
Live judge calls per visitor session / per day. |
JUDGE_DEMO_MAX_INPUT_CHARS / _MAX_ROWS / _MAX_CANDIDATES |
4000 / 25 / 5 |
Demo input caps. |
JUDGE_DB_PATH |
.llm_judge/experiments.db |
Experiment-tracking database: a SQLite file, or a postgresql:// URL such as Supabase (needs the postgres extra). Passwords are redacted wherever it's shown. |
JUDGE_MAX_DATASET_BYTES |
50000000 |
Larger dataset files are rejected before parsing. |
JUDGE_MAX_DATASET_ROWS |
100000 |
Maximum rows per dataset. |
JUDGE_BATCH_MAX_WORKERS |
4 |
Concurrency for batch mode (doubled when parallel bias runs are on). |
LOG_LEVEL |
INFO |
Logging verbosity. |