← Reference · Nestor G Pestelos Jr
Artificial Intelligence · Quality
LLM Evaluations
Reference entry · last updated August 25, 2026
LLM evaluations, or evals, are repeatable tests that measure a language model or agent against explicit success criteria. An eval gives the system a task, records one or more trials, and applies graders to the transcript, final output, or resulting state. For an agent, the object under test is the model plus its instructions, tools, retrieval, memory, policies, and orchestration.[1]
Unit of measurement
A task defines inputs and success criteria. A trial is one attempt. A grader scores an aspect of that attempt. A transcript or trace records the path. The outcome is the final state of the environment. These units matter because an agent can claim that it issued a refund while the payment system shows no refund. The state change, not the sentence, decides whether the journey succeeded.[1]
Stochastic systems need repeated trials. One successful run does not establish reliability. Report the number of successful trials over the number attempted, and keep the test conditions and versions with the result.
Journey-based evaluation
A conversational agent should be evaluated across a complete user journey, not only one reply. A guest-access journey can begin with “the door code does not work” and end with entry restored, a safe escalation, or an unresolved stay. Intermediate steps include identity checks, retrieval, tool calls, policy decisions, clarification, and host handoff.
Start with high-volume, high-cost journeys. Build cases from production failures, ordinary successful requests, policy boundaries, ambiguous requests, tool failures, and adversarial attempts. Anthropic recommends realistic tasks from observed failures and notes that 20 to 50 early tasks can be enough to start learning.[1]
Useful metrics
| Metric | Question answered | Required denominator or check |
|---|---|---|
| Journey success rate | Did the guest reach a correct end state? | Successful journeys / eligible journeys |
| Safe resolution rate | Was the request resolved without a policy or safety violation? | Safe successful journeys / eligible journeys |
| Escalation rate | How often did the agent hand off? | Escalated journeys / eligible journeys |
| Escalation precision | Did escalations need human judgment? | Necessary escalations / all escalations |
| Missed escalation rate | How often did a human correct an unsafe or unsupported action? | Late overrides / journeys that required escalation |
| Guardrail block rate | How often did the control plane reject a proposed action? | Blocked proposals / policy-checked proposals |
| Adversarial success rate | Did a malicious request cause prohibited disclosure or action? | Successful attacks / attack trials |
Raw counts are operational signals, not capability measures. Counts without denominators confuse traffic growth with worse behavior. Segment results by journey, property, channel, language, policy version, model, and risk tier. Do not optimize one number in isolation: lowering escalation can increase fabricated answers.
Capability and regression suites
A capability eval asks what the system can learn to do. It should include hard cases and may start with a low pass rate. A regression eval protects behavior that already works and should remain near 100 percent. When a capability becomes reliable, its cases can graduate into the regression suite.[1]
Run the relevant regression suite when a prompt, policy, retrieval source, tool contract, memory rule, model, or routing rule changes. Version the suite with those dependencies. A green model-only benchmark cannot certify an agent whose context or tools changed.
Graders and calibration
Use deterministic graders for facts that software can verify: a reservation exists, a refund amount matches policy, the correct property record was read, or a prohibited tool was not called. Use rubric-based model graders for qualities such as clarity or unsupported claims. Use humans to calibrate subjective graders and adjudicate difficult cases.[1]
No grader type covers every failure. Combine outcome checks, transcript checks, model rubrics, adversarial cases, and sampled human review. NIST describes testing, evaluation, verification, and validation as part of operational AI risk management across the system lifecycle.[3]
Production evidence
Offline evals test known cases before release. Production monitoring finds new distributions and failures after release. Convert verified production failures into eval tasks, then keep them in the regression suite. This creates a loop: observe, classify, reproduce, fix, evaluate, release, and monitor.
Pair automatic metrics with regular trace review. OpenAI describes evals as structured tests for variable AI systems and advises combining metrics with human judgment. Anthropic recommends automated evals, production monitoring, A/B tests, user feedback, transcript review, and periodic human calibration as complementary layers.[1][2]
Common measurement failures
Do not treat the agent's final sentence as proof of an outcome. Do not use only easy cases, only average scores, or only model-level benchmarks. Do not change the test set until a desired result appears. Do not let an uncalibrated model grade itself without checks.
Safety metrics also need severity. One disclosed door code can matter more than many harmless refusals. Report the failure class, exposure, affected journey, and whether the control plane blocked the attempt before an external action occurred.
See also
References
- ↑ Anthropic, “Demystifying evals for AI agents,” January 9, 2026. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- ↑ OpenAI, “Evaluation best practices.” https://developers.openai.com/api/docs/guides/evaluation-best-practices
- ↑ NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf