A Content Pipeline That Doesn't Trust Itself
written by Stefan Christoph
Two failures I’d rather not repeat
In July I open-sourced the core of my content pipeline (the machine behind the three-posts-a-week workflow) as a standalone repo [1]: ideation to final draft, quality gates included. Since then the private version kept evolving, and this weekend I synced the public repo up to it.
The honest way to explain what’s new is to explain what went wrong.
The first failure. A post of mine cited a source for a biographical detail. The claim-verification gate checked it and wrote “Verified” into its report. A second, independent gate report agreed. The post shipped. The cited page said the opposite of what my post claimed. Both gates had done the same thing: they fetched the source, saw that it was about the right topic, and pattern-matched that into support. Neither report quoted the relevant sentence, or showed that it supported the claim.
The second failure. A post about a product launch stated an availability claim: a count of launch regions. The review gate confirmed it: “Yes — verified.” It was wrong. The gate ran on the same model that drafted the post, and a model reviewing its own output inherits its own blind spots. When I re-ran the same review on a different model, it caught the error immediately.
Neither of these is exotic. If you run an AI-assisted writing pipeline, versions of both are sitting in your future. The update ships the three gates I built in response.
Gate 1: Humanize without losing yourself
The most requested thing since the first release wasn’t fact-checking at all — it was the AI-smell problem: drafts that are factually fine but read like a model wrote them. The naive fix, “ask the LLM to humanize it,” fails in both directions at once. It misses mechanical tells, and it scrubs the quirks that make the writing yours.
The new humanize-pipeline skill splits the job into five layered stages:
Each stage does the one thing it’s actually good at. The scanner runs twice because in my runs, stage 2 kept re-introducing what stage 1 removed.
- A deterministic scanner builds a concrete fix list: em-dash counts, banned vocabulary, citation-markup artifacts, with line numbers. Regexes don’t get tired and don’t grade their own homework.
- A voice-calibrated LLM rewrite applies the fix list, calibrated against a
VOICE.mdprofile of your published writing. This is the part that matters: the profile legitimately overrides the generic humanizer rules where they conflict. If you naturally use em dashes, the rewrite keeps them. De-AI-ing a draft into a generic neutral register is a failure mode, not a success. - The scanner runs again, as the exit gate. In my runs, the voice rewrite kept re-importing soft AI vocabulary (the “remarkable”-class words), so the deterministic check has to come after the LLM, not just before.
- An independent model critiques the substance (more on this below).
- Detector scores are logged as telemetry, never optimized. In my experience, editing text to chase a detector score degrades the prose — and I treat the scores as perishable, since detectors chase moving model generations.
The repo ships the voice profile as a fill-in template, pre-populated for a fictional example author so a fresh clone runs before you’ve configured anything.
Gate 2: A reviewer that isn’t the writer
Failure two has a boring, structural fix: the reviewing model must be a different model than the drafting one.
The release includes a small critic agent (deliberately minimal: no tools, no MCP servers, pure text critique) that you invoke with an explicit model override. It reads the exact prose that will ship and returns a numbered must-fix list, or NO MUST-FIX. To be precise about what this buys: a second model isn’t magically objective — it can share training data and provider habits with the first one. What it does is decorrelate the review, so writer and reviewer are less likely to miss the same thing. And because it has no tools, it critiques reasoning and internal consistency; establishing whether a claim matches its source stays with gate 3, which actually fetches things. For posts whose subject is a product launch, or anything with load-bearing technical claims, the pipeline treats this gate as mandatory.
The rule that took discipline to encode: if the independent review can’t run, the pipeline fails loudly. No silent fallback to a same-model pass presented as if the gate had passed. A skipped gate you know about is recoverable; a gate that quietly lied to you is how failure two happens.
Gate 3: “Verified” requires a quote
Failure one gets the simplest fix of the three. Any Verified verdict in a claim-verification report must now carry a verbatim quote from the fetched source that supports the claim. A verdict without a quote is treated as Unverified by everything downstream.
It works because of what it forces, not what it checks. Requiring an extract makes the report expose the specific evidence its verdict depends on — and that evidence is checkable: the quoted span either exists in the fetched source or it doesn’t, and a reviewer (or the second model) can compare claim against quote directly. A quote can still be cherry-picked or misread, so this doesn’t make verification errors impossible. It makes the failure I actually had — topic-match rubber-stamped as claim-match, twice, with nothing to audit — hard to repeat and easy to catch.
Still clone-and-run
None of this changed the deal from the first release. The repo stays standalone: Kiro CLI [3] plus stdlib Python, zero MCP servers, and a bundled synthesized example workspace so it runs end-to-end before you point it at your own notes. Everything personal (your paths, your voice profile, your internal-leak patterns, your aliases) lives in gitignored *.local.json config files that never ship.
Before pushing this release I ran the updated pipeline against its own bundled example, headless, start to finish: draft, all gates, the new humanize stages included. The pipeline that doesn’t trust itself also doesn’t get to skip its own tests.
What it still deliberately doesn’t do: publishing. The repo stops at a review-ready final draft. Deploy tooling, video derivatives, and social formats live elsewhere. The plan is separate, composable repos you plug together as you need them, rather than one monolith.
Credit where due: stage 2 builds on the open-source humanizer methodology [2]. The voice-calibration override and the surrounding gate structure are what this release adds around it.
A walkthrough video for this release is coming — Walkthrough video — link coming soon.
Your pipeline doesn’t need to be smarter. It needs to stop grading its own homework. Which of your gates would survive a reviewer that wasn’t the writer?
Sources
- [1] kiro-content-pipeline on GitHub — the release: humanize-pipeline, content-critique, the critic agent, and the hardening rules
- [2] blader/humanizer — the open-source humanization methodology stage 2 builds on
- [3] Kiro CLI — the reference agent runtime the repo targets
About the Author
Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.
This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.
❤️ Created with the support of AI (Kiro)