Observability for AI Agents
What to log and monitor in agentic systems: trajectories, tool I/O, costs, outcomes, and replay for debugging.
- observability
- operations
- production
Agent observability is the ability to reconstruct what an agent did, why it did it, what it cost, and whether it succeeded.
Without trajectories, you are debugging folklore.
The unit of observability: the run
Each run should have a stable run_id and:
- Trigger source (user, cron, webhook)
- Goal + constraints
- Policy/version identifiers
- Ordered events (model, tool, human, system)
- Final status: success / fail / escalate / budget
Event types worth storing
| Event | Key fields |
|---|---|
model_call |
model, tokens, latency, prompt hash |
tool_call |
name, args, result code, latency |
decision |
plan update, route choice |
policy |
allow/deny, reason |
human |
approval requested/granted |
outcome |
grader scores, user feedback |
Redaction
Never store raw secrets. Redact:
- API keys, passwords
- Payment payloads
- Unnecessary PII
- Credentials inside tool args
Keep enough structure to debug (field names, error classes).
Metrics dashboard (starter)
- Task success rate by intent
- p50/p95 latency
- Cost per successful task
- Tool error rate
- HITL rate
- Step count distribution
- Kill-switch / budget-stop rate
Replay and bisect
Good logs enable:
- Local replay with frozen tool stubs
- Diffing trajectories across prompt versions
- Finding first diverging step between good/bad runs
Privacy and retention
- Set retention by sensitivity
- Tenant isolation in log stores
- Access control for trace viewers
- Export paths for incident response
Summary
If you cannot answer “what tools ran, with what args, leading to what state?” you do not have agent observability. Instrument runs like distributed transactions—because that is what they are.
Frequently asked questions
What should you log for an AI agent?
Log the goal, model calls, tool inputs/outputs (redacted), decisions, costs, policy outcomes, and final task result so you can replay and grade the run.
Related reading
Agent Evaluation: How to Measure Reliability, Cost, and Success
A practical framework for evaluating agentic systems: task success, trajectories, tool correctness, cost, latency, and regression suites.
Common AI Agent Failure Modes
The most frequent ways agents fail in production: loops, tool misuse, hallucinated success, injection, and scope escape—and how to mitigate them.
AI Agent Production Readiness Checklist
A practical checklist for shipping agentic systems: tools, auth, budgets, evals, observability, HITL, and incident response.
Trajectory
A trajectory is the ordered record of an agent run: thoughts or decisions, tool calls, observations, and outcomes.