← Reference · Nestor G Pestelos Jr
Artificial Intelligence · Operations
AI Agent Observability
Reference entry · last updated August 25, 2026
AI agent observability is the ability to reconstruct and measure how an agent handled a request across model calls, retrieval, memory, tool use, policy checks, external state changes, and human handoffs. It extends distributed tracing with agent-specific evidence. It does not require storing a model's private chain-of-thought.
Trace structure
A trace represents one end-to-end operation. Spans represent work inside it: an inbound request, retrieval, a model call, a tool call, a policy decision, or an escalation. W3C Trace Context standardizes identifiers that let separate services preserve one trace across process and vendor boundaries.[1]
A user journey can outlive one request. Keep both a trace_id for each request and a separate journey_id or conversation identifier for correlated requests. OpenTelemetry defines a session as a collection of logs, events, and spans with a shared identifier.[2]
What to capture
| Stage | Minimum evidence |
|---|---|
| Request | Trace and journey IDs, timestamp, channel, property or tenant ID, classified intent, risk tier |
| Configuration | Agent, model, prompt, policy, router, and tool-contract versions |
| Context | Source identifiers and versions, retrieval scores, memory references, missing required fields |
| Model call | Provider, requested and returned model, operation, latency, token counts, structured output status |
| Tool call | Tool name and version, validated arguments or safe digest, result status, side-effect identifier |
| Control | Proposed action, policy rule, allow, block, clarify, or escalate decision, confidence or reason code |
| Outcome | Verified external state, final response class, latency, cost, escalation, override, user-visible result |
OpenTelemetry's generative-AI conventions define attributes for agents, conversations, models, operations, retrieval documents, tool calls, and token use. Several fields are marked as likely to contain sensitive information.[3]
Decision evidence, not hidden reasoning
“Capture the agent's thinking” is too broad. Store inspectable decision evidence: what the system knew, which rule applied, what action it proposed, what validator returned, and why the control plane allowed or stopped it. Use stable reason codes such as policy_missing, identity_unverified, tool_timeout, and refund_above_limit.
Private chain-of-thought is not a reliable audit contract. It can expose sensitive data, vary by model, and differ from the actual control decision. A trace should make behavior reproducible without requiring hidden free-form reasoning. If a human-readable explanation is needed, generate a concise decision summary from the structured evidence and label it as a summary.
Outcomes and overrides
Record the state after the agent acts. A tool returning HTTP 200 does not prove that the guest entered the property or received a refund. Link the trace to a durable action receipt, reservation event, payment event, access-system event, or host resolution.
Human handoff is part of the trace, not an error outside it. Record when the escalation happened, its reason, what context was shown to the host, the host's action, whether that action overrode the agent, and the final outcome. These fields support escalation precision, missed-escalation analysis, and policy improvement.
Privacy and security
Telemetry can become a second database of guest messages, access codes, personal information, and tool credentials. Collect only what the operational question requires. Redact or tokenize identifiers, separate sensitive payloads from indexed attributes, limit access, encrypt data, set retention by data class, and audit access.
OWASP advises against directly logging session identifiers, access tokens, passwords, encryption keys, sensitive personal data, and other primary secrets. It recommends masking, hashing, sanitizing, or encrypting such data when an operational need exists.[4] OpenTelemetry also warns that model messages, retrieval queries, and tool arguments or results can contain sensitive information.[3]
Metrics derived from traces
Traces support journey success, safe resolution, escalation, override, policy block, tool failure, clarification, latency, and cost metrics. Keep the numerator and denominator. Segment by journey, risk, property, model, policy version, and channel.
Sample ordinary successful traces for cost control, but retain or preferentially sample policy blocks, escalations, overrides, tool errors, adversarial attempts, and negative outcomes. A sampling policy must not erase the rare events that safety analysis needs.
Common failures
Logs without correlation IDs cannot reconstruct a journey. Traces without version identifiers cannot explain regressions. Raw prompts without retrieved-source IDs cannot diagnose stale context. Tool-call success without outcome verification measures plumbing, not user success.
Observability is also not an enforcement layer. It explains what happened. A separate control plane must block unsafe actions before they reach tools. Monitoring that notices a leaked door code after delivery is evidence of a failure, not prevention.
See also
References
- ↑ W3C, Trace Context, Recommendation, November 23, 2021. https://www.w3.org/TR/trace-context/
- ↑ OpenTelemetry, “Semantic conventions for session,” development status. https://opentelemetry.io/docs/specs/semconv/general/session/
- ↑ OpenTelemetry, “Generative AI attributes,” semantic conventions registry. https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/
- ↑ OWASP Cheat Sheet Series, “Logging Cheat Sheet,” section “Data to exclude.” https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html