TL;DR: Most teams ship LLM judges without testing them against human labels. The result: judges that are confidently, consistently wrong on 30%+ of cases. Hamel Husain’s “critique shadowing” methodology (pass/fail + written critiques from a domain expert) builds trustworthy judges. Amazon Bedrock Model Evaluation handles model comparison at scale. You need both — plus periodic human review to catch drift.

You would not ship a feature without testing it. You would not deploy a model without benchmarking it. But most teams ship their LLM judges without verifying them at all.