schristoph.online

Tag

LLMEvals

4 posts · all tags · all posts

2026

  1. The 6x Math Jump Looks Like a Loop Fix

    TL;DR: An MIT CSAIL team reported that GPT-6 Astra edited 0.016% of Qwen3.5-0.8B’s weights and lifted its MATH-500 score from 7.9% to 48.1%, with GSM8K and IFEval rising too. The last column of the …

    EvaluationLLMEvalsLLMLimitationsReasoningVerificationAmazonBedrock

  2. Wrong, Not Broken: An Agent Evaluator Is Only as Good as Its Reference

    TL;DR: Matt Wood’s “Wrong, not broken” describes a refund agent that answers in 600 ms with zero errors and quotes $40 where the policy says $140, because it retrieved last quarter’s document [1]. …

    ObservabilityAgenticAILLMEvalsAIQualityOpenTelemetryCloudWatch

  3. Who Referees the AI?

    TL;DR: In July 2026 a purported disproof of the Collatz conjecture got through Lean’s official proof checker, and through an independent second checker as well. It was wrong. The submission exploited …

    AIAgenticAILLMEvalsVerificationSecurityFormalMethods

  4. Your AI Judge Needs a Judge

    TL;DR: Most teams I see ship LLM judges without testing them against human labels. The result: judges that are confidently, consistently wrong on 30%+ of cases. Hamel Husain’s “critique shadowing” …

    AILLMEvalsAmazonBedrockAIQualityAgenticAIMachineLearning