Reliability safeguards

Reliability safeguards

Safeguard	What it does
Neutral schema	The prompt shows the output shape with placeholders, never example scores or winners that could anchor the judge.
Untrusted-content boundaries	The task and outputs sit between markers that carry a random per-request token. The judge is told to treat them as data and to ignore embedded instructions.
Exact criteria matching	The reply must contain every requested criterion exactly once (case-insensitive). Missing, extra or duplicate criteria are rejected, never guessed.
Repair retries	A reply that is malformed or doesn't match the schema is sent back with the reason it was rejected, up to JUDGE_MAX_PARSE_RETRIES times.
Brace-aware recovery	JSON embedded in stray prose is recovered with a real JSON decoder, not string slicing.
Scale normalisation and weights	Each score is mapped to 0-1 on its own scale before the weighted overall score is computed.
Consistent bias merges	With mitigation on, explanations come from the ordering that agrees with the averaged verdict. When neither ordering agrees, both are kept and labelled.

Results also carry warnings (they never change scores):

Code	Meaning
possible_prompt_injection	An output contains text aimed at manipulating the judge.
possible_verbosity_bias	The winner is much longer than the other output (JUDGE_VERBOSITY_RATIO).
self_preference_risk	A candidate model is from the judge's own model family.
position_inconsistent	The verdict changed when the outputs were swapped.
score_out_of_range	The judge scored outside the scale; the score was clamped.
input_truncated	An input exceeded JUDGE_MAX_INPUT_CHARS and was truncated.
provider_fallback	The primary provider failed and a configured fallback (JUDGE_FALLBACK_PROVIDERS) answered instead.

These signals are heuristics. A clean result doesn't prove the absence of bias or manipulation, and a warning doesn't prove its presence.