AI Agent Production Readiness Checklist
A practical checklist for shipping agentic systems: tools, auth, budgets, evals, observability, HITL, and incident response.
- checklist
- production
- operations
Use this checklist before exposing an agent to real users, real data, or real side effects.
Product and scope
- Top user goals documented with success criteria
- Explicit non-goals (what the agent must refuse)
- Owner team and on-call rotation defined
- UX for progress, failures, and escalations designed
Architecture and tools
- Tool list minimized; each tool has an owner
- Least-privilege credentials per tool
- Irreversible tools gated (HITL or dual control)
- Timeouts, retries, and idempotency keys defined
- Staging tools cannot touch production data
Safety and policy
- Tenancy enforced in code, not prompts
- Spend and step budgets enforced
- Injection threat model for untrusted content
- Kill switch tested
- Secret/PII redaction in logs
Evaluation
- Offline suite covers top intents (30+ tasks minimum)
- Automatic graders for critical outcomes
- Adversarial/policy cases included
- Suite runs in CI on relevant changes
- Cost and success tracked per release
Observability
- Full trajectories stored (redacted)
- Metrics: success, latency, cost, tool errors
- Alerts for spend spikes and error storms
- Replay path for debugging
Operations
- Runbook for bad actions and partial failures
- Versioned prompts/models/configs
- Canary or percentage rollout plan
- Backup human process if agent is disabled
Go / no-go
Ship only if: success is measurable, blast radius is bounded, and you can stop the agent fast.
Related deep dives: Safety, Evaluation, Observability.
Frequently asked questions
When is an AI agent ready for production?
When you can authenticate callers, constrain tools, measure task success, cap cost, audit actions, and roll back or disable the agent quickly.
Related reading
Agent Evaluation: How to Measure Reliability, Cost, and Success
A practical framework for evaluating agentic systems: task success, trajectories, tool correctness, cost, latency, and regression suites.
Agent Safety and Guardrails
Practical guardrails for agentic AI: permissions, sandboxes, human approval, budgets, content filters, and audit trails.
Agentic Architecture: Core Components of Production Agents
A practical architecture model for agentic systems: goals, planners, tools, memory, orchestration, evals, and control planes.
Observability for AI Agents
What to log and monitor in agentic systems: trajectories, tool I/O, costs, outcomes, and replay for debugging.