schristoph.online

Tag

AIQuality

4 posts · all tags · all posts

2026

  1. Golden Questions: Every Agent Change Is a Release

    TL;DR: An agent is a model, a prompt, a set of tools and the data behind them. The model, prompt, tool contracts, policies, schemas and the pinned test data each change on their own schedule, and each …

    AIAIAgentsEvaluationAIQualityAgentCoreVerification

  2. Wrong, Not Broken: An Agent Evaluator Is Only as Good as Its Reference

    TL;DR: Matt Wood’s “Wrong, not broken” describes a refund agent that answers in 600 ms with zero errors and quotes $40 where the policy says $140, because it retrieved last quarter’s document [1]. …

    ObservabilityAgenticAILLMEvalsAIQualityOpenTelemetryCloudWatch

  3. A Content Pipeline That Doesn't Trust Itself

    TL;DR: My AI content pipeline once recorded a claim as “Verified” in two separate gate reports while the cited page said the opposite — both gates had matched the page’s topic, not the claim. Another …

    AIContentAIAgentsAIQualityVerification

  4. Your AI Judge Needs a Judge

    TL;DR: Most teams I see ship LLM judges without testing them against human labels. The result: judges that are confidently, consistently wrong on 30%+ of cases. Hamel Husain’s “critique shadowing” …

    AILLMEvalsAmazonBedrockAIQualityAgenticAIMachineLearning