Agent Evaluation: How to Measure Reliability, Cost, and Success

Agent evaluation measures whether an agent completes real tasks correctly, safely, and economically—not whether a single response “sounds good.”

Chatbots can be scored on answers. Agents must be scored on outcomes and trajectories.

What to measure

1. Task success (primary)

Define success in checkable terms:

• Ticket correctly categorized and required fields filled
• PR opened with tests passing
• Refund issued only when policy allows
• Research brief includes required sources

Prefer automatic graders (assertions, scripts, DB checks) over vibes.

2. Trajectory quality (secondary)

Even when the final answer is right, inspect the path:

• Unnecessary tool calls
• Repeated failed actions without adaptation
• Skipped verification steps
• Leaked secrets into tool args

See trajectory.

3. Tool correctness

Score each tool invocation:

• Valid schema?
• Correct tool selected?
• Arguments faithful to evidence?
• Errors handled?

4. Safety and policy

Track:

• Disallowed tool use
• Scope escape (accessing wrong tenant/data)
• Missing human approval on gated actions
• Hallucinated permissions (“I deleted it” when tool failed)

5. Cost and latency

Per task and per step:

• Tokens in/out
• Tool runtime
• Wall-clock time
• Cost to successful completion (not only cost per run)

An agent that succeeds 20% more often at 3× cost may still be a win—or not. Product decision.

Evaluation layers

Layer	When	Purpose
Unit	On tool/adapters	Schema, mocks, permissions
Component	Planner, router, memory	Local decision quality
Scenario / task	Offline suite	End-to-end outcomes
Online	Production	Drift, real failures, cost
Human review	Sampling	Rubrics, edge cases, UX

Building an offline suite

1. Collect real tasks from logs, support, or internal dogfood
2. Anonymize sensitive data
3. Write grader functions (preferred) or rubrics
4. Freeze tool simulators or sandboxes for determinism where possible
5. Version the suite with your agent release

Example grader sketch

def grade_refund_agent(run) -> dict:
    ok_policy = run.final.state["refund_allowed"] == run.expected["refund_allowed"]
    ok_amount = run.final.state.get("amount") == run.expected.get("amount")
    no_extra = "chargeback" not in run.tools_called
    return {
        "pass": ok_policy and ok_amount and no_extra,
        "scores": {"policy": ok_policy, "amount": ok_amount, "scope": no_extra},
    }

LLM-as-judge: when it helps

Use judges for:

• Open-ended writing quality
• Citation faithfulness
• Conversation tone

Do not use judges alone for:

• Financial correctness
• Permission boundaries
• Anything with a ground-truth state transition

If you use a judge, fix the rubric, model, and temperature; measure judge agreement on a labeled sample.

Metrics that matter

Quality

• Task success rate
• Partial credit / stepwise success
• Human escalation rate
• Critical failure rate (wrong irreversible action)

Efficiency

• Median steps to success
• Cost per successful task
• p95 latency

Reliability

• Flake rate across seeds/models
• Tool error recovery rate
• Regression delta vs previous release

Regression workflow

Treat agents like software:

1. PR opens → run smoke suite (fast)
2. Merge to main → full suite
3. Canary in production → online monitors
4. Scorecards per release: success, cost, safety

Never “prompt tweak to prod” without at least a smoke suite for top intents.

Online evaluation

Offline suites lag reality. Add:

• Outcome labels from users (thumbs, ticket reopen, edit distance)
• Implicit signals (task completed in host system)
• Trace review queues for low-confidence runs
• Alerts on spend spikes and tool error storms

Common evaluation mistakes

• Scoring only the final natural-language answer
• No sandbox → flaky live tools as “evals”
• Suites full of toy demos, empty of real edge cases
• Optimizing demos that never appear in production traffic
• Ignoring cost until finance notices

Related: Common agent failure modes.

Minimal starter plan (one week)

Day 1–2: Define 20 tasks with automatic graders
Day 3: Log trajectories for current agent
Day 4: Score success/cost/safety
Day 5: Fix top 3 failure clusters
Day 6–7: Wire suite into CI smoke

Summary

Evaluate agents as goal-seeking systems: outcomes first, trajectories second, economics always. If you cannot state success criteria in code or a rubric, you are not ready to claim the agent works.