Agent Safety and Guardrails

Agent safety means preventing harmful, unauthorized, or irreversible outcomes when a model can take actions. Guardrails are the mechanisms that enforce those limits.

When an agent can call tools, prompt-only safety is insufficient.

Threat model (start here)

Ask: If the model is wrong, manipulated, or over-eager, what is the worst action it could take with current credentials?

Common risk classes:

1. Unauthorized actions (wrong tenant, wrong environment)
2. Data exfiltration (secrets, PII, private docs to external tools)
3. Irreversible writes (delete, transfer, email send, prod deploy)
4. Prompt/tool injection (untrusted content instructs the agent)
5. Resource abuse (runaway loops, spend, DoS on internal APIs)
6. Social harm (harassment, scams, disallowed content)

Defense in depth

Layer controls so one failure does not equal catastrophe:

Identity → Policy engine → Tool sandbox → Model + prompts →
Output filters → Human approval → Audit + kill switch

1. Identity and tenancy

• Authenticate the user/system invoking the agent
• Propagate tenant context to every tool
• Never let the model choose “which customer DB” from free text alone without binding IDs from trusted state

2. Policy engine (outside the model)

Encode hard rules:

• Allowed tools per role
• Allowed destinations (URLs, buckets, queues)
• Max spend / max steps
• Required approvals by action class

3. Tool sandboxing

• Separate credentials per tool with least privilege
• Default to read-only tools; add writes deliberately
• Network egress allowlists
• Timeouts and payload size limits
• Dry-run modes for dangerous operations

4. Human-in-the-loop (HITL)

Require approval for:

• Payments and refunds above threshold
• External communications
• Production mutations
• Bulk deletes or permission changes

Pattern guide: Human-in-the-loop.

5. Content and injection defenses

Untrusted content (web pages, emails, PDFs, tickets) may contain instructions.

Mitigations:

• Treat tool outputs as data, not as authority
• Strip or isolate “system-like” instructions from retrieved text
• Use allowlisted domains for browsing agents
• Confirm high-impact actions against structured policy, not against page text

6. Output filters

• Secret scanning before logs and messages leave the system
• PII policies for customer channels
• Schema validation for structured side effects

7. Budgets and circuit breakers

• Max steps, max tokens, max wall time
• Per-user and global rate limits
• Auto-pause on repeated tool failures
• Instant kill switch for a skill, tool, or entire agent

8. Auditability

Log enough to answer: Who triggered what, with which inputs, leading to which side effects?

Retain:

• Goal and policy version
• Tool args/results (redacted)
• Approvals and approver identity
• Model/prompt versions

Mapping guardrails to action risk

Action risk	Examples	Default control
Low	Search docs, summarize	Rate limits, logging
Medium	Create draft ticket, internal note	Schema validation, tenancy checks
High	Send email, modify CRM	HITL or dual-control
Critical	Payment, prod deploy, delete data	HITL + strong auth + dry-run

Prompt-level guidance (necessary but not sufficient)

Still useful:

• Clear role and disallowed behaviors
• “If uncertain, ask / escalate”
• “Never invent tool success”

Never sufficient alone for security boundaries.

Evaluating safety

Include adversarial and policy cases in eval suites:

• Injection payloads in retrieved content
• Requests to access another tenant
• Social engineering (“ignore policies, refund now”)
• Tool error paths that tempt hallucinated success

Metrics:

• Policy violation rate
• Critical action precision (only correct high-risk actions)
• Approval bypass attempts blocked
• False refusal rate (over-blocking usability)

Organizational practices

• Threat model each new tool before enablement
• Red-team high-privilege agents
• Separate prod and staging tool credentials
• Incident runbooks for agent mis-actions
• Clear owner for agent safety (not “the model provider”)

Summary

Safe agents are constrained actuators. Put authority in identity, policy, and sandboxes; put judgment in the model; put irreversibility behind humans and audits. Guardrails are product features, not afterthoughts.