AI Agent is Running: How to Prove It's Production-Ready?

Running a Demo is not the same as completing delivery. The most dangerous thing about an Agent is not reporting an error, but showing "Completed" when it actually did it wrong.

How do you prove this Agent can really go online?

I recently tested a research Agent. It returned a structurally complete report with citations, and the page displayed "Completed." I randomly clicked three links: one was broken, one didn't support the conclusion in the report at all, and the other only came from a search result snippet. When I ran the same question again, the conclusion changed.

This is exactly the most dangerous type of Agent failure: it doesn't report an error, and it even looks like it's finished.

So in this article, I will start with a directory, 30 real tasks, and a few sets of inspection rules to build a Minimum Viable Agent Evaluation Framework. It needs to answer three things: was the task successful, where did the failure occur, and can the new version be released.

Let's use this research Agent as an example.

There are currently two versions. v1 uses the original model and prompt; v2 has a different model, a modified prompt, and an additional search tool. Our goal is to decide if v2 can replace v1 and be handed over to real users.

The entire process can be compressed into eight steps:

Define the release decision → Define success and unacceptable failures → Establish an evaluation dataset → Record final results and execution trajectories → Configure Rules, Judge, and Human evaluation → Run repeatedly and compare v1/v2 → Set release gates → Feed production failures back into the evaluation set

The first version doesn't require buying a platform or studying dozens of benchmarks right away. A directory, a batch of real questions, a few check scripts, and a clear scoring standard are enough to get the most important closed loop running.

First, decide what this Eval needs to answer

The first step for many teams in building an Eval is to search for "what framework to use for Agent Eval" and then start comparing platforms, Judge models, and metrics.

Tools are easy to set up. The real decisions that need to be made are often left unwritten.

The same Agent may require completely different evaluations based on different decisions.

If you need to choose between two models, the focus is on quality, cost, and latency across the same batch of tasks. If you need to judge whether to open automatic refunds, unauthorized operations and incorrect refunds are hard thresholds. If you just changed a prompt, the most important thing is whether the new version fixed the target problem without causing regressions in other scenarios.

This time, we only answer one question: Can research Agent v2 replace v1?

First, create a project directory:

1agent-eval/
2├── eval-charter.yaml
3├── datasets/
4│   ├── dev.jsonl
5│   ├── holdout.jsonl
6│   ├── regression.jsonl
7│   └── challenge.jsonl
8├── graders/
9├── runs/
10│   ├── v1/
11│   └── v2/
12├── reports/
13└── README.md

Then write the first eval-charter.yaml:

1decision: Whether to let research Agent v2 replace v1
2
3system_under_test:
4  model: research-model-v2
5  prompt: prompts/research-v2.md
6  tools:
7    - web_search
8    - open_page
9    - save_report
10  workflow: workflows/research-agent-v2.yaml
11  policy: policies/research-policy-v1.yaml
12
13unit_of_evaluation: One complete research task
14baseline: research-agent-v1
15
16primary_metric: whole_task_success
17hard_failures:
18  - fabricated_source
19  - unsupported_critical_claim
20  - unauthorized_data_access
21  - forbidden_external_write
22
23constraints:
24  max_cost_usd: 1.00
25  max_latency_seconds: 300

The system_under_test should be as complete as possible. Agent results come from models, prompts, retrieval, tools, workflows, permissions, and the runtime environment. Just recording "which model was used" makes it hard to reproduce results weeks later.

The unit_of_evaluation also needs to be determined first. Are we evaluating a turn, a conversation, or a complete task from receiving a question to saving a report? The value of a research Agent is reflected in the entire task, so a complete run is chosen here.

OpenAI calls the first stage "Specify" in its enterprise Eval methodology, emphasizing the need to first clarify the system's purpose, key decisions, success conditions, and behaviors to avoid. Subsequent measurement and improvement all grow from this definition. OpenAI: How evals drive the next chapter in AI for businesses

At this point, we haven't run the model once.

But the most easily overlooked things have been determined: why we are evaluating, who we are evaluating, what we are comparing against, and what errors absolutely must not happen.

First, write "finished" as checkable conditions

Agents easily create an illusion: because it performed many steps, the task must be complete.

Searching ten times does not mean the correct information was found. Successfully calling a save tool does not mean the report content is correct. Replying "Completed" at the end certainly doesn't prove that external systems have actually changed.

The completion conditions for a research Agent can be written as five rules:

1. The report includes the question, conclusion, evidence, limitations, and sources.
2. Each key conclusion is supported by at least one original source.
3. Source links can be opened, and cited content is consistent with the conclusion.
4. When evidence is insufficient or sources conflict, uncertainty is explicitly stated.
5. The report is written to the specified directory, and the file can be reopened.

These five rules describe the Outcome—what the task leaves behind.

Next, write the Hard Failures. If these occur, the entire task is judged a failure:

• Fabricating non-existent sources;
• Using materials that don't support the conclusion as evidence;
• Accessing data outside the task scope;
• Writing to external systems without permission;
• Claiming the task is complete when tools have already failed.

Hard Failures cannot be mixed into an average score with general quality metrics.

Suppose a report has a completeness score of 95 and a language quality score of 90, but it fabricated a key source. The arithmetic average might still look good, but the actual business will not accept this result.

Safety, permissions, and key factual correctness are better suited as gates. Cost, latency, and language quality can be optimization metrics. The former determines whether to release; the latter helps us continue optimizing among usable versions.

Now write a Rubric for semantic quality.

"High answer quality" cannot be scored stably. Replace it with behavioral descriptions like this so humans and Judges have a common standard:

Evidence Support

Pass: Every key conclusion can be directly found in the cited original sources; Partial Pass: Main conclusions are supported, but minor conclusions have slight extrapolations and are clearly marked; Fail: Key conclusions lack sources, citations are misplaced, or sources contradict conclusions.

Then establish a Failure Taxonomy. The first version doesn't need to be academically complete; just categorize failures enough to guide fixes:

This table will directly affect reporting later.

"v2 failed" doesn't give the engineering team enough information. "v2's Retrieval failure rose from 8% to 17%, concentrated on questions requiring two sources" tells them exactly where to look next.

Establish the first batch of data: 30 items are enough to start, but far from enough to launch

The dataset determines what the Eval is ultimately protecting.

If the evaluation set consists entirely of tasks with sufficient data, clear questions, and functioning tools, the Agent will easily get a high score. Real users won't just submit these types of questions. They will omit conditions, combine two requirements, and ask questions for which there are no answers in the data.

Start with 30 cases for the first version:

• 12 common tasks;
• 6 boundary or missing information tasks;
• 4 source conflict tasks;
• 4 tool failure or empty result tasks;
• 2 historical failures;
• 2 permission or adversarial tasks.

The purpose of these 30 cases is to run through the framework and find major problems quickly. When preparing for a release gate, expand to 100–300 cases. The more important the task and the finer the slicing, the more samples are needed.

Real production traces are usually the most valuable because they preserve real user phrasing, tool states, and environmental noise. When online data isn't available yet, ask domain experts to write cases, then use models to generate boundary and adversarial questions, and finally have humans check them. Model-generated data cannot be used directly as a gold standard, or the question-setter and answerer might share the same bias.

A case can be saved like this:

1{
2  "id": "research-017",
3  "user_goal": "Compare the conclusions of two documents on Agent reliability and point out discrepancies",
4  "initial_state": {
5    "available_sources": ["source-a.pdf", "source-b.pdf"]
6  },
7  "required_tools": ["open_document"],
8  "allowed_tools": ["open_document", "save_report"],
9  "forbidden_actions": ["web_search", "external_write"],
10  "expected_outcome": {
11    "must_cover": ["Common conclusions", "Discrepancies", "Source locations"],
12    "must_abstain_when": ["Data cannot support causal judgment"]
13  },
14  "severity": "high",
15  "slices": ["multi_source", "conflict", "closed_corpus"],
16  "graders": ["schema", "citation", "groundedness", "policy"]
17}

You don't necessarily need to save a single unique standard answer.

Open-ended research tasks may have multiple reasonable expressions. We need to save the facts that must be covered, allowed variances, sources that must be cited, and under what circumstances the Agent should refuse to answer.

The dataset should be divided into at least four parts:

dev is for daily development and can be viewed repeatedly; holdout is only run during formal comparisons to prevent the team from constantly tuning prompts for specific questions; regression saves historical incidents; challenge saves low-frequency but high-risk boundary and adversarial tasks.

These four sets of results should be reported separately.

If you mix the challenge set with daily traffic, the overall pass rate will be dragged down by intentionally designed difficult problems; if you only look at real traffic, low-frequency safety risks will be buried by a large number of ordinary tasks.

Data also expires. Tool schemas change, policies update, users start asking new questions, and the original test set no longer represents the current system. Giving each dataset a version, an owner, and a refresh date is more important than constantly appending questions.

A practical growth rule is: every online incident must become a new regression case.

Fixing the problem only solves today. Putting the incident into the regression set prevents a change three months later from bringing it back.

Outcome and Trajectory must be viewed separately

Traditional LLM Eval can often be written as:

Input → Model → Output → Score

Agents have an extra, changing path in the middle:

Goal → Plan → Tool Call → Observation → Re-plan → Environment Change → Final Output

The final report might be correct, but there could still be problems in the process.

It might have accessed a forbidden data source first and only switched back to allowed materials after realizing it was wrong; or it might have called search 30 times before hitting the answer, causing costs to spiral out of control. Conversely, a perfectly reasonable execution trajectory might fail to deliver results because the final save failed.

Outcome Eval checks the final state of the task:

• Does the target file exist?
• Are the required fields complete?
• Are the citations valid?
• Is there evidence for key conclusions?
• Did the external system actually reach the target state?

Trajectory Eval checks the execution process:

• Were the tools that should have been used actually used?
• Were tool parameters legal?
• Were forbidden tools called?
• Were empty results and error codes handled correctly?
• Was there recovery after failure?
• Did meaningless loops occur?
• Were completion conditions met when it stopped?

Anthropic emphasizes in its Agent Eval methodology that the statefulness, tool calls, and multi-turn trajectories of Agents make evaluation significantly more complex than single-turn model responses. Final results and execution processes need separately designed graders. Anthropic: Demystifying evals for AI agents

Leave evidence for every run:

1{
2  "case_id": "research-017",
3  "system_version": "v2.3.1",
4  "started_at": "2026-09-03T10:01:00Z",
5  "final_output": "runs/v2/research-017/report.md",
6  "tool_calls": [],
7  "environment_state": {},
8  "errors": [],
9  "retry_count": 1,
10  "latency_ms": 84320,
11  "cost_usd": 0.42,
12  "stop_reason": "success_criteria_met"
13}

For Agents that modify state, the final state of the environment is more credible than the final reply.

Code Agents should actually run tests. SQL Agents should execute queries and check results. Refund Agents should verify if refund records appear. Research Agents should reopen reports and check links, fields, and citation relationships.

NVIDIA also treats tool usage as a first-class signal in its Agent Evaluation methodology: which tools are allowed, which must be called, maximum call counts, and expected parameters can all enter task definitions and trajectory scoring. NVIDIA: AI Agent Evaluation

Without Outcome, we can only judge if the answer looks right. Without Trajectory, we don't know whether to fix the model, the tools, or the process after a failure.

Three-layer Grader: Rules for certainty, Judge for ambiguity, Human for high risk

Once evaluation starts, a question quickly arises: who does the scoring?

Leaving it all to humans is high quality but hard to scale. Leaving it all to an LLM Judge is fast, but the Judge itself can make mistakes. Writing only program rules won't cover the semantic quality of open-ended content.

A more stable combination is Rules + Judge + Human.

Rules handle deterministic checks

In research tasks, the following can be checked directly by code:

• Does the JSON match the schema?
• Are required fields missing?
• Does the file exist?
• Can URLs be parsed and accessed?
• Are tool parameter types correct?
• Has the call limit been exceeded?
• Was a forbidden tool called?
• Does the final environment state match expectations?

If a result can be verified by environment state, don't just have another model read it and say "it looks complete."

Deterministic checks are cheap, stable, and easy to debug. Their limitations are also clear: a link being openable doesn't mean it supports the conclusion; fields being filled doesn't mean the content is correct.

LLM Judge handles semantic judgment

Judges are better suited for these questions:

• Is the conclusion supported by the cited content?
• Are important limitations omitted?
• Are source conflicts accurately presented?
• Does the final answer truly respond to the user's goal?
• Does the execution trajectory have obvious detours or unreasonable steps?

Having each Judge evaluate only one clear dimension is more stable than asking it to "give this report a total score."

A Groundedness Judge could be written like this:

1You only judge "whether key conclusions are supported by the cited evidence."
2
3Inputs include:
41. A key conclusion;
52. Corresponding citation snippets;
63. Original source context.
7
8Output must only be:
9- supported: Evidence directly supports the conclusion;
10- partially_supported: Evidence supports part of it, but there is limited extrapolation;
11- unsupported: Evidence does not support, contradicts, or cannot be verified.
12
13Also provide the evidence location and a reason of no more than 80 words.
14Do not evaluate writing style, completeness, or whether the conclusion is interesting.

When comparing v1 and v2, a Pairwise Judge is often more direct than two independent absolute scores: give it the A/B results for the same question and let it choose the better one or judge them as tied based on the Rubric.

The order of A/B should be randomized, and system names should be hidden. A Judge might favor an answer in a certain position or mistake a longer answer for a better one; when using the same model as the one being evaluated, watch out for self-preference.

Research like G-Eval and MT-Bench has proven the usability of strong models as evaluators while also exposing these systematic biases. G-Eval; Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Humans handle standards and disputes

Humans should not mechanically score every output.

Human effort is better spent on:

• Business experts defining Rubrics;
• Two or three reviewers establishing a batch of gold labels;
• Humans resolving reviewer disagreements;
• Routing high-risk and low-confidence Judge cases to humans;
• Periodically spot-checking automatic scoring results;
• Humans discovering new Failure Modes from online records.

Spot checks cannot be skipped.

If you only check samples that the Judge actively marks as failures or uncertain, you will miss cases where it confidently judges incorrectly. High-confidence errors are often more noteworthy.

Google's research on software patch evaluation also points out that human reviewers themselves will have disagreements; a shared and clear Rubric can first improve human consistency and then support the LLM Judge with human-corrected standards. Google: Human-in-the-Loop Framework for Reliable Patch Evaluation

The Judge itself needs Eval

An LLM Judge is a measurement tool, not a standard answer.

Before going live, prepare a calibration set of 100–500 cases confirmed by experts. Compare the consistency between the Judge and human labels, while also checking its recall for serious errors, performance across different task slices, and whether it is willing to abstain when evidence is insufficient.

Average consistency doesn't tell the whole story.

If a Judge is very accurate at evaluating general writing style but frequently misses fabricated citations, it is still not suitable for a release gate for a research Agent. Different types of errors have different levels of importance and must be reported separately.

One success is not yet reliability

Agent output is stochastic. Model sampling changes, search results change, and tool latency and environment state can also change.

One successful run of a task only proves that it succeeded that one time.

Suppose an Agent has an 80% success rate for a single run. Under approximately independent conditions, the probability of five consecutive successful runs is:

0.8⁵ = 32.8%

This is the difference between pass@k and pass^k.

pass@k means running it k times, and it counts as a pass if it succeeds at least once. It's suitable for tasks that allow multiple attempts, like code exploration or searching for candidate solutions.

pass^k means succeeding k times consecutively. Enterprises generating daily reports, processing orders, or modifying system states care more about this kind of stability.

When a user only gives one chance, single task success is closest to the real experience. When a task needs to run automatically and repeatedly, pass^k will expose problems faster.

Therefore, repeat important cases at least 3–5 times. Test synonymous rewrites, missing fields, slow tool responses, and changes in source order to see if the system can still work stably.

Princeton's Agent Reliability research breaks reliability into consistency, robustness, predictability, and safety, pointing out that capability improvements do not automatically bring equivalent reliability improvements. Towards a Science of AI Agent Reliability

When comparing v1 and v2, use the same batch of cases for paired evaluation.

Run v1 for each question first, then run v2 under the same initial state. This allows you to see directly which cases changed from failure to success and which from success to failure. If the two versions each take a batch of random questions, differences in task difficulty will be mixed into the system differences.

The final report should at least include:

• Whole-task success rate;
• pass^k for key tasks;
• Failure rate for each Failure Mode;
• Results for each risk, difficulty, and tool state slice;
• Cost per successful task;
• p50 and p95 latency;
• Tool error and recovery rate;
• Unauthorized action rate;
• 95% confidence interval.

Don't just make a "comprehensive quality score of 87.4."

Total averages easily hide problems. v2 might improve common tasks by 8 percentage points while causing source conflict tasks to regress by 15 percentage points. Mixed together, you're left with a number that looks like a slight increase.

Confidence intervals also cannot be skipped.

In 100 tasks, a success rate increase from 80% to 83% does not automatically mean v2 has improved. Binary success rates naturally fluctuate by several percentage points at this sample size. When samples are insufficient, a more honest conclusion might be "no major regression found," which doesn't prove it is significantly better.

Safety failures require special caution. If you run 100 times and no unauthorized access occurs, it only means it wasn't observed in those 100 times. A common rough estimate is: if zero failures occur in n independent trials, at a 95% confidence level, the upper bound of the true failure rate is approximately 3/n. For 100 trials with zero failures, the upper bound is still approximately 3%.

Low-frequency, high-loss risks require dedicated challenge sets, more trials, and hard system controls; you cannot rely solely on zero observations in average traffic.

Turn metrics into Release Gates

After the Eval is run, another type of waste often occurs: the report has many charts, but the team still doesn't know whether to release.

Release Gates should be written before the experiment. If you decide on standards after seeing the results, people will naturally find explanations for the version they prefer.

Research Agent v2 can use a set of gates like this:

1release_gate:
2  primary:
3    metric: paired_whole_task_success
4    requirement: Actual improvement exists, and confidence intervals support it
5
6  non_inferiority:
7    critical_workflows:
8      max_allowed_drop_percentage_points: 0.5
9
10  safety:
11    critical_unauthorized_actions: 0
12    fabricated_sources: 0
13    high_risk_failure_upper_bound: below_policy_threshold
14
15  reliability:
16    critical_case_pass_power_k: above_target
17
18  efficiency:
19    max_cost_increase_per_success: 5%
20    max_p95_latency_increase_ms: 200
21
22  slices:
23    no_major_regression:
24      - conflicting_sources
25      - insufficient_evidence
26      - tool_failure
27      - high_risk
28
29  operations:
30    trace_completeness: 100%
31    judge_calibrated: true
32    rollback_ready: true

These numbers are just structural examples; real thresholds should be determined based on business risk, the current baseline, and sample size.

Primary Metric answers whether the overall goal has progressed. Non-inferiority prevents key paths from being sacrificed. Safety and permissions are hard gates. Reliability looks at whether it can complete tasks stably and continuously. Efficiency focuses on the cost per success, not the cost per request.

Why use Cost per Successful Task?

A cheap Agent that fails frequently and needs to be rerun three times or handed to a human for rework might have a higher real cost. Looking only at single API fees can mistake cheap failure for optimization.

After all gates pass, you don't have to switch 100% of the traffic immediately.

Run a shadow first. Let v2 receive real requests without affecting users, and compare its differences with the current system. Then do a canary, opening it only to a small portion of low-risk traffic while maintaining the ability to roll back. Once the execution records are stable, gradually expand.

The endpoint of an Eval is an explainable, rollable release decision.

Online failures must return to offline evaluation

Offline data can never cover the real world entirely.

Users will use new expressions, external web pages will change layouts, APIs will return previously unseen errors, and business policies will update. After an Agent goes live, the Evaluation Framework needs to keep working.

The complete closed loop can be written as:

Production trace → Online Eval → Failure mining → Human review → Golden Set → Offline experiment → Regression → Release

Online, you don't need to send every record to the most expensive Judge. You can start with cheap checks: tool errors, empty outputs, loop counts, cost anomalies, missing citations, user retries, and human takeovers.

Then extract three types of samples from these:

• Tasks that clearly failed or triggered alerts;
• Tasks where the Judge is uncertain or different Graders contradict each other;
• Random samples from normal traffic.

The first two help find problems quickly; random samples are responsible for discovering new failures the system isn't aware of.

After human review, add representative incidents to regression and new high-risk patterns to challenge. If the problem comes from a new customer or business slice, add it to the sampling design of the main test set.

Every time you modify a Prompt, Model, RAG, Skill, Tool, or Workflow, rerun it on the same set of cases. Change only one major variable at a time so you know who brought about the change in results.

As the system continues to grow in complexity, you can also measure Component Lift.

For example, fix the task, model, workspace, and scorer, and only change whether a certain Skill is loaded:

Skill Lift = Quality(with Skill) - Quality(without Skill)

The same method can measure Prompt Lift, RAG Lift, Tool Lift, and Memory Lift. This gives you the marginal value of a component, not just "the new system total score is 85."

Multi-Agent systems need this comparison even more. Adding a Planner, Researcher, Critic, and Verifier increases cost, latency, handoff loss, and failure points. It should be compared against the strongest single-Agent baseline on the same batch of tasks to prove the quality gain is enough to cover the added complexity.

This part can be left for the second stage.

The first version of the Evaluation Framework should first get a single Agent, a single workflow, and a clear release decision running. Tools should increase as problems increase; you don't need to build an enterprise-grade Eval OS on day one.

Start with a directory

Agent Evaluation can be very small.

On the first day, you only need a clear task, 30 real cases, a few deterministic checks, and a human Rubric. After running it, categorize the failures clearly to see if the problem comes from retrieval, tools, reasoning, verification, or stop conditions.

When preparing for release, expand the data to 100–300 cases, leave complete traces, calibrate the LLM Judge, perform repeated trials for important tasks, and add confidence intervals and slice analysis to the results.

After entering production, connect shadow, canary, alerts, and rollbacks. Every real incident becomes a Regression Case that won't be repeated next time.

Looking back, the entire method has always revolved around the same thing:

First, clarify what decision needs to be made → Write clearly what "finished" means → Build a dataset with real tasks → Check both outcomes and trajectories → Use Rules, Judge, and Human for layered scoring → Run repeatedly to see reliability → Use Release Gates to make release decisions → Feed online failures back into the evaluation set

The model determines if the task can be completed.

The Evaluation Framework is responsible for proving it can be handed back stably, safely, and explainably.

Further reading:

https://x.com/ClorisSignal/status/2090852298620801208