schristoph.online

Observability · 13 min read

Wrong, Not Broken: An Agent Evaluator Is Only as Good as Its Reference

Stefan Christoph▶ Watch the walkthrough
On this page
TL;DR: Matt Wood’s “Wrong, not broken” describes a refund agent that answers in 600 ms with zero errors and quotes $40 where the policy says $140, because it retrieved last quarter’s document [1]. Quality evaluators can catch that kind of failure, but which evaluator you pick matters. I rebuilt the run with a small agent on Amazon Bedrock and scored it with Amazon Bedrock AgentCore Evaluations. The built-in faithfulness, helpfulness and correctness evaluators all rated it well, because the answer is faithful to the wrong document. Correctness failed it once I passed the expected $140 answer. A short version rule on the retrieval span failed it too. In Amazon CloudWatch Omni those map to a built-in evaluator with a ground-truth expected response and a custom code-based evaluator [3] [4]. Record the document version on the retrieval span so an evaluator has something to compare.

When I spoke at a Grafana event in Munich in September, one slide bridged from dashboards to agents with a “wrong, not broken” beat. The phrase, and the example behind it, come from Dr. Matt Wood’s AWS Insights post of the same name [1]. An agent answers in 600 milliseconds, the error rate is zero, and it tells a customer they are owed $40 when the policy says $140. The document it retrieved was last quarter’s version. “Every dashboard is green. Nothing appears to be broken. It’s wrong.” [1]

The post argues that agent observability needs a quality signal next to latency and errors, and that teams have to make “good” explicit enough to measure [1]. I agree with it. What the essay doesn’t answer is which evaluator would actually have caught this run, so that is where I’ll pick the refund example up.

The run that looks perfect

I had built a small version of that refund agent earlier and traced it with the AWS Distro for OpenTelemetry. For this post I built an even smaller one, with the Strands Agents SDK and Claude Haiku 4.5 on Amazon Bedrock, in US East (N. Virginia). It has one tool, retrieve_policy. The index behind that tool still holds the stale 2026-Q2 policy, which gives a $40 credit for a late delivery. The 2026-Q3 policy, which applies to the request, says $140. I captured the agent’s OpenTelemetry spans and ran it twice, on 13 October 2026. The code is in a small sample folder [18].

This is what the first run reports:

{
  "answer": "Based on our refund policy, you are owed a $40 credit for your late delivery. We also cover return shipping if you need to return any items.",
  "spans": ["chat", "execute_tool retrieve_policy", "execute_event_loop_cycle",
            "chat", "execute_event_loop_cycle", "invoke_agent Strands Agents"],
  "error_spans": 0,
  "wall_clock_s": 2.01,
  "retrieved_version": "2026-Q2"
}

The second run gave the same answer in 2.06 seconds. Nothing in the first four fields signals a failure, even though the answer is wrong. The trace is nested, the agent called the tool it was told to call, nothing threw, and it was quick. The only hint is the last line, and it’s there only because I chose to put the document version on the tool span. Error count and duration, two of the signals behind request, error and duration metrics [2], look healthy here, which is the point of the original example [1]. A single trace can’t show request rate.

Four evaluations and one rule on the same trace

I sent the captured spans to the on-demand Evaluate API of Amazon Bedrock AgentCore Evaluations, which is the service CloudWatch Omni uses to score agents [3] [16]. I sent them straight from memory, where the getting-started guide downloads them from CloudWatch Logs first [16]. Four evaluations ran on the trace, three with only the trace and one with the expected answer. Both runs got the same scores from Builtin.Faithfulness, Builtin.Helpfulness and Builtin.Correctness:

Built-in evaluatorExpected answer givenScore
Faithfulnessno1.0, “Completely Yes”
Helpfulnessno0.83, “Very Helpful”
Correctnessno1.0, “Perfectly Correct”
Correctnessyes, $1400.0, “Incorrect”

Then I ran a rule of a few lines of Python against the same spans. It reads policy.version from the tool span and compares it with the version that applies to this request, here 2026-Q3. It fails.

Diagram: a run with status OK goes from a question to retrieval of the stale 2026-Q2 policy to the answer 40 dollars. Faithfulness, which compares answer and retrieved document, passes. Correctness with the expected 140 dollars fails. A version rule that compares the document version with the version applicable to the request, here 2026-Q3, fails.

The same trace scored three ways. Only the checks that know the truth fail it.

The faithfulness judge explained its score in one line: “All stated facts align with the tool output.” And that’s true. The failure happened upstream, in retrieval, and the model did its job well. It read the context and repeated it accurately, and any check that only looks at the answer and the context it was given sees a consistent story.

This is uncomfortable, because grounding checks are among the first evaluators teams add, and for a good reason, they catch hallucination. A faithfulness evaluator is defined as checking “whether the response is faithful to the provided context” [5]. A stale document is exactly the case where the provided context is the problem. By its definition the check should score this run well, and it did.

Without a reference, the correctness judge noticed the policy’s effective date, called the missing caveat “a completeness issue rather than a factual error”, and gave full marks. With the expected answer its explanation got short: “This is a critical factual error - the dollar amount is significantly different ($40 vs $140).” Two runs are not a statistical sample, and LLM judges can score the same trace differently on another day. But both runs scored the same, and the result follows from what each evaluator is given.

What CloudWatch Omni gives you for this

CloudWatch Omni, generally available since September 23, scores agent responses with evaluators, “continuously in production, and on demand against a dataset”, and puts the scores on the same trace as latency, errors and token usage [2] [9]. The evaluators run through Amazon Bedrock AgentCore Evaluations. The catalog has three kinds: built-in, third-party metrics from the DeepEval and AutoEval libraries, and custom evaluators [3]. Matt Wood’s post and the launch post both put the number of built-in evaluators at 17, covering criteria such as correctness, faithfulness and tool selection [1] [12].

Two parts of that catalog fit the refund case. The built-in correctness evaluator accepts ground truth. You pass an expectedResponse for a trace, and it “measures how accurately the agent’s response matches the expected answer” [4]. Without it, the same page says, the evaluator “still works without expectedResponse, it just evaluates based on context alone” [4]. So the same evaluator name can mean two different checks, depending on whether you hand it the truth. In my run it scored 1.0 without the reference and 0.0 with it.

Custom evaluators come in three forms, and one of them is code, “your own AWS Lambda function that receives the trace content and returns a score”, meant for “deterministic, domain-specific checks” [3]. That’s the one I’d use here. A freshness check is that kind of rule. Read the version from the retrieval span, look up which version applies to the request, return pass or fail. That lookup matters, because a refund for an order placed last quarter may correctly fall under last quarter’s policy, and a rule that always demands the latest version would fail those runs and teach the team to ignore it. There’s no judge model involved, so as long as the evaluator code and its dependencies are versioned, policy versions are never edited in place, and the lookup is pinned to the request’s date, the same trace gets the same score.

On general criteria, the Omni documentation itself says “an agent can pass all of them while doing the wrong thing for your domain”, and tells you to write a custom evaluator “for every rule that is specific to your agent” [3]. Its tutorial uses a support agent that answers an off-topic cookie-recipe question, and general-purpose evaluators rate the answer as helpful [6]. The stale refund fails the same way. In the tutorial the prompt was missing a rule. In my run the agent answered correctly from an out-of-date document.

Where the reference comes from

The stale index itself is a data-pipeline bug, and the real fix belongs there, which is to index each policy version with its effective date and retrieve the one that applies to the request. The evaluator is the backstop for the day that fix fails. If the evaluator scores that traffic, a stale document then lowers a score you’re watching before a customer complains.

The obvious objection is that in production you don’t have an expected answer for a live question. If you did, you wouldn’t need the agent.

You need one less often than it seems, and offline the reference is easy to get. Every bad run you find becomes a candidate test case once you establish, from the source of record, the answer it should have given, and Omni’s loop is built around that: you turn failing traces into a dataset, fix the agent against it, then keep scoring live traffic [6]. Matt Wood puts it this way: “The traces where an agent got something wrong can become a dataset.” [1]

Online, it’s usually enough to know the facts the answer depends on. For a refund agent that’s the policy version applicable to the request and the amount it states. Those live in a system you already own, so a code evaluator can look them up at scoring time. Ground truth here is a lookup against the source of record.

A judge model still has a place, for criteria nobody can turn into a rule, such as tone or whether an escalation was appropriate. A judge can use a reference too, since custom trace-level evaluators accept an {expected_response} placeholder next to the conversation and the response under evaluation [4]. Judges need checking too. The documentation advises testing a custom judge on a known-bad and a known-good trace before you rely on it [3], and I wrote earlier about calibrating judges against human labels [13]. A reference-based check is easier to trust because you can verify the reference.

Putting the document version on the span

The freshness check only worked because the retrieval span carried the document version. The tutorial is blunt about it: “An evaluator cannot score content that was never recorded.” [6]

Two settings decide what your traces contain. The first is content capture. The Omni telemetry guide sets AWS_GENAI_CONTENT_EXTRACTION_OPT_OUT=true to keep prompt and response text in the spans, which the trace view and the evaluators read. For Node.js agents the guide says this content is captured by default [7]. The second is yours, the attributes you add to your own spans. Framework instrumentation records model and tool calls. It doesn’t know that your retrieval tool returned version 2026-Q2 of a policy unless you tell it. So add the document id, version and effective date as span attributes. That is a few lines of code, and it turns “was this answer right?” into a question a rule can answer.

The same attributes help when you investigate. Matt Wood’s post asks “Why did refund accuracy drop for European customers this week? Did anything change in retrieval…?” [1]. With versions on the spans, you can group the failing traces by the document they read.

Omni next to Grafana

Omni is built on OpenTelemetry and on the CloudWatch data stores, so your existing CloudWatch dashboards and queries keep working unchanged [2]. Telemetry you already send to CloudWatch appears in it “with nothing to reconfigure”, and it’s “not a replacement” for anything you run today [2]. On the Grafana side, the Amazon CloudWatch data source has a PromQL query type for the OpenTelemetry metrics you ingest into CloudWatch [8]. That was also the point of my Munich slide: instrument once with OpenTelemetry, keep Grafana for the operational view, and pick up the quality scores where the evaluations run.

Omni is generally available in US East (N. Virginia), US West (Oregon) and Europe (Ireland), and the FAQ still listed those three in mid-October [9] [10]. Evaluations are charged at Amazon Bedrock AgentCore Evaluations rates, separately from ingestion, storage and query [11], and the Omni pricing page’s production-agent example samples 2% of runs [11].

Built-in evaluators are billed by tokens, at $0.0024 per 1,000 input tokens and $0.012 per 1,000 output tokens [15]. With the token counts AgentCore’s own pricing example assumes (15,000 in, 300 out), one built-in evaluation comes to about four cents [15]. Custom evaluators are $1.50 per 1,000 evaluations, with any model usage billed separately [15]. A code-based check calls a Lambda function and no judge model, so as I read the price list it costs $0.0015 per evaluation plus Lambda request and duration charges and whatever the lookup costs.

My own traces were much smaller than the pricing example. The four built-in evaluations of one run used about 3,500 input and 700 output tokens together. At those rates that’s about two cents for the run, so under half a cent per evaluation. The code check’s $0.0015 fee is about a third of that on my traces, and about a twenty-sixth at the pricing example’s size, before Lambda and lookup charges. How much more traffic that buys depends on your trace size and Lambda duration.

Where I would start

Pick one agent that makes decisions with consequences.

  1. Make sure its traces are nested and contain input and output. Before you keep that content in production, redact what you don’t need at capture time and limit which members can read the rest [7] [17]. Also add the version of every document or record it retrieves as a span attribute.
  2. Keep the general-purpose evaluators you have. They catch a different kind of failure.
  3. Write one rule for the fact that would hurt most if it were stale or wrong, including which version applies to which request, and run it as a code-based evaluator.
  4. When you find a bad run, save it with the answer it should have given. That collection becomes your regression dataset.
  5. Watch the pass rate of the rule over time. A score that drops while latency and errors stay flat is the “wrong, not broken” signal.

One caution from an earlier post: once a score becomes a target, teams optimize the score [14]. A rule tied to the source of record is harder to game than a general quality grade, but it still measures only what you wrote down.

Which fact in your agent’s answers would you most want a rule for?

Sources

🎬 Also available as a blog walkthrough video on YouTube

❤️ Created with the support of AI (Kiro)

Related

AIYour AI Judge Needs a JudgeAIWho Referees the AI?AgenticAICan You Trust a Swarm's Reasoning?WeeklyReviewWeekly Review — Oct 26–Nov 1, 2026

ObservabilityAgenticAILLMEvalsAIQualityOpenTelemetryCloudWatch