Skip to content

RAG, agents and reliability

RAG, agents and judge reliability

RAG evaluation

result = judge.evaluate_rag(
    "What is the range of the Aurora 3?",
    contexts=["The Aurora 3 has a range of up to 80 km.", "Showroom hours: 10-18."],
    answer="Up to 80 km, and it comes with a free helmet.",
    reference="Up to 80 km.",  # optional: enables context recall
)
result.faithfulness  # 0.5: share of the answer's claims supported by the contexts
result.unsupported_claims  # ['It comes with a free helmet.']: possible hallucinations
result.context_precision  # average precision of the relevant contexts, in rank order
result.context_recall  # share of the reference covered by the contexts
result.answer_relevance  # how directly the answer addresses the question

The judge returns claim-level verdicts and the metrics are computed from them, so every number can be traced back to specific claims and passages. Retrieved passages are fenced as untrusted data and scanned for prompt injection. CLI: llm-judge rag --question ... --context ... --answer ....

Agent evaluation

judge.evaluate_agent(trace) takes a framework-agnostic trace: the task, the available tools, the ordered steps (thought, tool_call, tool_result, message) and the final answer. It returns two separate assessments:

  • Deterministic checks. Unknown tools, missing required arguments, malformed calls, failed tool results, repeated identical calls, step-budget overruns, expected tools never used, and a missing final answer.
  • A judged grade on Task Completion, Tool Selection and Step Efficiency (configurable), optionally against a reference answer.

CLI: llm-judge agent --trace-file trace.json.

Judge reliability

  • measure_consistency(judge, task, output_a=..., output_b=...) judges the same input repeatedly. It reports agreement with the modal verdict, the spread, and a bootstrap 95% CI. At temperature 0 this only shows provider nondeterminism, so raise JUDGE_TEMPERATURE to probe sensitivity. CLI: llm-judge consistency ... --samples 5 --temperature 0.7.
  • judge_agreement(store, run_a, run_b) compares two recorded runs over the same rows, typically by two different judges. It uses Cohen's kappa for winners, and Spearman correlation plus mean absolute difference for scores. CLI: llm-judge experiments agreement <run-a> <run-b>.

All intervals come from repeated samples. None of them is a model's self-reported confidence or a calibrated probability.