The Gate That Couldn't See the Defect

If you have ever shipped something that passed CI and broke anyway, this is that story with unusually clean numbers.

What this is about, briefly

I narrate demo videos in a synthesised clone of my own voice. The mechanism matters for one reason only: to clone a voice, you hand the model a short reference clip of someone speaking, and it produces new speech that carries that clip’s character. The clip is the input that determines quality, and it is easy to get wrong.

Because it is easy to get wrong, I built a small tool that grades a reference clip before you use it: signal-to-noise, clipping, loudness, duration, speaking rate, pitch variation, pauses. It gives each take a score, a star rating, and a verdict of keep, review, or reject [1].

Then I recorded three new reference takes for the walkthrough, and did not run it on any of them.

The narration came back with audible crackling. That part is easy to laugh at, and it isn’t the interesting part. The interesting part is what happened when I finally did run the tool, and what the other two checks were doing while all this went on.

The three principles

If you read nothing else, these are the transferable part. Each one cost me a round of rework.

A check only sees the defect class it measures

This sounds too obvious to state, and I still got caught by it. The fix is not more sophisticated checks. It is asking, before you trust one, which defect class is this blind to? — and answering it empirically rather than by reasoning, by feeding the check artifacts whose real outcome you already know.

A signal that doesn’t gate is a signal you’ll scroll past

Detection and decision are separate problems, and getting detection right buys you nothing if the verdict blends the signal away. Every condition you detect needs an explicit action attached — block, quarantine, or force a human look. Three things set which: how bad the failure is, how much you trust the detector, and how expensive a false positive is to recover from. That third one is what licenses a conservative block on thin evidence. A note nobody reads is the one option that is always wrong.

An uncalibrated metric is a hypothesis, not evidence

Before you act on a new measurement, run it on a case you know is good and a case you know is bad. If it does not separate them, you do not have a measurement yet. You have a direction, and it may be the wrong one.

The rest of this post is the evidence for those three, and then a fourth lesson that isn’t about checks at all.

The check that ran and couldn’t see it

Each narration segment goes through an automated check: synthesise it, transcribe the result back to text, compare that transcript to the script. High similarity means the voice said what it was supposed to say.

A reviewer flagged a trailing non-lexical sound at the end of one segment, the kind of vocal shrug that makes a sentence sound unfinished. I went to find it in the check’s output.

It had scored 1.00. Not “high”. Perfect.

Which is exactly right, and completely useless. Transcription compares words, and the defect was a sound with no words in it. If the recognizer simply omits that sound, the transcript comes back identical to the script and the similarity is 1.00 by construction — so the metric cannot reliably detect this defect class at all. I had been treating a full score as evidence the segment was clean.

The replacement took twenty minutes: measure the energy envelope over each segment’s final 250 milliseconds and flag anything still loud where a sentence should have decayed. It surfaced the reported artifact and three more I hadn’t noticed, all four confirmed by ear.

It is a screening heuristic, not a detector. A sustained final phoneme, a breath, or room reverberation can legitimately keep that window energetic, and I picked the window and threshold by hand on a handful of segments rather than validating them against labelled examples. Its value isn’t precision. It is that it perceives a different defect class than word comparison does, and I had only built one of the two.

The check I skipped, which would have passed it anyway

Now the reference clip. The take behind the crackling scored keep, four stars, with the label too quiet sitting right there in its output.

Both halves are correct in isolation. The reference passed the tool’s checks for background noise, clipping, and frequency range, at 45 dB of signal-to-noise ratio, the voice sitting about 45 decibels above the room’s hiss, which is comfortably clean. And it was genuinely quiet. But the verdict blended those into a pass, because I had decided loudness was a matter of degree rather than a hard failure. Clipping rejects. Noise rejects. Being too quiet just got a note.

Here is what that note was hiding, and it takes three numbers to make sense of it.

Loudness gets measured in LUFS, Loudness Units relative to Full Scale: a perceptual measure of programme loudness rather than of the tallest peak, referenced to digital full scale, so real recordings come out as negative numbers. The measurement algorithm behind it is specified by ITU-R BS.1770, which names the unit LKFS [5]; LUFS is the equivalent term used by EBU R 128 [6], and the two are numerically the same. Either way it is a different quantity from peak level, which is measured in dBFS. My pipeline normalizes every reference to −16 LUFS.

This take sat at −38 LUFS. Reaching −16 meant about 22 dB of gain, and decibels are logarithmic, so that is not a nudge: it multiplies the waveform by roughly twelve and a half times. The reference I ended up shipping sat at −25 LUFS and needed 9 dB, a multiplication of about 2.8.

Amplification is indiscriminate. It multiplies the voice and everything underneath it by the same factor: the room, the preamp hiss, and the recording’s quantization error. These references are 16-bit integer PCM, so every sample is rounded to one of 65,536 representable levels, and under ideal rounding without clipping that error is bounded by half a level. It is not strictly a steady hiss: undithered, the error correlates with the signal rather than behaving as independent noise, which is the whole reason dither exists [7]. But its ceiling is set by the bit depth and does not move when you record more quietly. So a quieter take has less margin above it, needs more gain to reach the target, and brings more of it up on the way.

How much of the crackling that explains, I genuinely do not know. Gain does not worsen the floor’s ratio to the signal, quantization error is only one unmeasured candidate alongside room and preamp noise, and I measured none of them separately. What follows is the shape of the hypothesis, not a result.

The working explanation is that the clone, a self-hosted model on Amazon SageMaker [2], is conditioned on a reference whose own floor has been dragged up toward audibility, and it reproduces the character of what it is conditioned on. What I can say with confidence is narrower: the two takes near −38 LUFS produced clearly audible crackling on listening, and the −25 LUFS reference of the same voice did not. Those takes also differed in text and room, so this is a correlation across a handful of recordings, not a controlled ablation.

The failure I can be firm about is not detection. Detection worked, and the label was specific and correct. It was that a condition with a hard downstream consequence had been encoded as a soft preference. Four stars reads as “fine”, so I scrolled past it. Twice.

The metric that separated nothing

Told the segments sounded “not properly stitched”, I reached for the obvious measurement: sample-level discontinuity. Count the places where consecutive audio samples jump further than they plausibly should. Rough, but it should separate a clean join from a torn one.

I ran it on the version that had just been rejected: 502 jumps. Then, out of habit rather than rigour, on an earlier version that had been accepted: 503.

Two raw counts, nearly equal. No rates, no repeated runs, no test, so I won’t call it a statistical result. But as a screen for “which of these two has the stitching problem”, it discriminated nothing, and I had been about to spend an afternoon on the answer it implied. I never established that this count tracks anything a listener notices, so I stopped treating it as evidence in either direction and went back to listening.

The leading explanation for what reviewers were hearing is not a torn join at all. To slow the delivery down I had been inserting a fixed silence between sentences. Measured against a single continuous generation of the same words, my version produced 6 pauses averaging 0.73 seconds where the model’s own prosody produced 11 averaging 0.35, fewer and longer and deader gaps from roughly the same total pause time. That is a plausible cause of “audible breaks”, and removing it removed the complaint, but a single before-and-after that also changed how the audio was generated does not isolate the pause strategy as the cause.

Where these principles apply outside audio

None of the three depends on anything about speech, which is why they are worth stating separately from the story.

The blind-spot principle is the one behind a passing benchmark that fails to predict production behaviour. An exact-match score that reads 1.00 on an answer that is technically correct and operationally useless is the same failure as a transcript score reading 1.00 on an audible artifact: the metric measures a proxy, the proxy is silent about the defect class you care about, and a high number gets read as quality. The mitigation is the same too. Hold out artifacts whose real outcome you know, and confirm your metric ranks them correctly before you trust it to rank anything else.

If you already run mutation testing, you have the instinct. Mutation testing asks whether your tests fail when you deliberately break the program. The blind-spot question asks whether your gate fails when you deliberately hand it a bad artifact. Same move, aimed at measurement rather than at code. I had been doing the first for years and had never once thought to do the second.

The gating principle shows up wherever a warning has no consequence attached. A linter rule nobody blocks on, an alert that pages nobody, a dashboard metric with no threshold: each is a signal you have correctly detected and decided not to act on, and the decision is usually implicit rather than chosen. Warn-versus-block is a design decision that deserves the same thought as the detection logic, and it should follow all three factors above: consequence, how much the detector has earned your trust, and what a false alarm actually costs to undo.

The calibration principle applies to any metric you are about to reason from, especially one you just invented under pressure while debugging. That is precisely the moment you skip validating it, and precisely the moment a wrong direction is expensive.

The fourth lesson, which isn’t about checks

Chasing a calmer delivery, I asked for three reference takes at progressively slower speaking rates. All three were futile, and one division would have told me so before I asked.

The script was 798 words and the target runtime was five minutes. The pace being asked for was around 150 words per minute, a measured, unhurried read. Divide: 798 ÷ 150 is 5.3 minutes, or 5:19. The script by itself overran the target at the pace everyone wanted, which means no delivery change could have satisfied both. Something had to give, and it was never going to be the arithmetic. Cutting to 523 words brought it to 3:29 at that pace, with room to spare.

Here is what five references produced. Each one was used to synthesize the same test line with the same settings, so the text is held constant; reference rate is what I read, output rate is what came back:

Every take came out faster than it went in, by 45 to 79 wpm, and the deliberate 78 wpm read still landed at 151. Note what that does and does not show. It rules out reading slower as a way to make the output slower by the same amount, which is what I had assumed. It does not rule out some noisier relationship being predictive with more data. What I can say is that across these five trials in this pipeline, reference pace gave me no usable control over output duration, and that was the only thing I needed to know before stopping.

I had a fitted line here in an earlier draft, complete with a slope, an intercept and a correlation coefficient. Preparing the table above is what killed it: the numbers I could still verify do not reproduce the line I had been quoting, in this post or in my own internal notes. Five points was never enough to support that precision anyway, and I have removed the equation everywhere rather than publish a coefficient I cannot derive from data I can show you. It is an uncomfortable illustration of the third principle, arriving late and at my own expense.

What did move the timing was punctuation in the synthesis text. On the line I tested, commas lengthened it by about 25% and produced real silence, while ellipses and extra full stops added none that I could measure — a single line and one set of settings, so treat it as a local observation. And the largest lever on the finished runtime was not delivery at all: cutting 275 words out of the script.

The recording effort was wasted, but the waste wasn’t the recording. It was asking for it before doing the arithmetic that showed it couldn’t work. Before you ask someone for effort, check that the effort can succeed. That one is not about voice cloning either.

Everything that changed, and where

Six rejected versions produced one shipped video and a longer list of changes than I expected. Several are corrections to guidance I had written myself and would otherwise have kept following.

The reference tool [1] got two thresholds recalibrated against clips whose clone quality I could already judge. The tool’s preferred duration band went from 12–35s to 10–20s, and too quiet is now a hard reject below −33 LUFS instead of a four-star pass. Neither number is an established optimum. The band is the tool’s provisional preference, narrowed because the old one graded two references that produced worse clones as ideal, and the model card does not speak to an upper bound at all.

The −33 cut deserves its own defence, because by my own second principle a hard block on two correlated examples is over-confident. Those examples motivated the cutoff; they did not validate it, and I have no false-positive rate for it. What justifies blocking anyway is the third factor above: a false positive costs about two minutes of re-recording, and a false negative cost me six rounds of review. With that asymmetry I would rather reject a usable take than pass an unusable one, and I would rather say that out loud than pretend the number is validated. Treat that number as a conservative pipeline policy rather than a validated causal threshold: it is a provisional cut chosen so the two takes correlated with audible crackling fall below it while a third that produced a merely worse clone stays a warning. Two positive examples can separate a threshold; they cannot validate one.

The tool also gained a command-line entry point, because the whole argument of this post is that you should grade your references and the only way to do that used to be a web UI:

python -m voice_studio.score take1.wav take2.wav        # full scorecard
python -m voice_studio.score --quiet --no-transcript *.wav   # one line each

Its exit code is the worst verdict across all files — 0 keep, 1 review, 2 reject — so it works as a gate in a pipeline rather than only as something to read.

Beyond that: synthesis now runs one call per segment rather than one per sentence, which removed the invented inter-sentence timing described above and, as a side effect I had not attributed to it, stopped short lines coming out rushed. The narration check gained the envelope test. And the writing checks for this blog gained a rule of their own: an early draft of a paragraph above opened “The argument was that…”, which states a claim at arm’s length instead of making it. My prose scanner had no pattern for that family, so it passed clean and a human reader caught it. It now flags it, with guards so that referring to an argument as an object stays legal, and tests so nobody deletes the pattern for being noisy [3].

Two corrections went into my own documentation, both found by having an independent model review this post rather than by any gate. My internal guidance credited the 10–20 second reference band to the model’s vendor; the model card says nothing of the sort, and I had propagated that invented attribution into five files [4]. The same guidance described the encoded reference as “a couple of hundred few-shot examples”, which is not what audio codec frames are.

Some of this is in tooling you cannot see, and it is worth being explicit about which. Unpublished: the trailing-artifact envelope check, the one-call-per-segment synthesis change, a duplicate of the level check inside my synthesis script, and the corrections to my own internal guidance documents. Public and inspectable: the reference tool, with both recalibrated thresholds and the new CLI [1], and the prose scanner with its tests [3]. If you want to reproduce anything here, those two are the ones you can actually run.

That duplicate deserves a note, because it is the same mistake one layer up. The right place for a reference-quality check is the tool whose entire job is grading references, where anyone can use it. Putting a copy in my private synthesis script was convenient for me and useless to everyone else. The public gate is the one that counts.

Five of those changes are checks or corrections to checks, and not one of them was found by the check it belongs to.

Sources

• [1] voice-sample-studio — the reference grading tool, its thresholds, the new CLI, and a self-test. Background on why reference quality matters: Your Voice Clone Is Only as Good as the Reference Clip
• [2] From a Generic Voice to My Own: Self-Hosting a TTS Model on Amazon SageMaker — the endpoint the narration runs on
• [3] kiro-content-pipeline — the writing checks, including the summary-frame detector and its tests
• [4] Qwen3-TTS — model card; the Base model is described as cloning from three seconds of reference audio, and does not specify an upper bound
• [5] ITU-R BS.1770 — the recommendation defining the loudness and true-peak measurement algorithms, in LKFS
• [6] EBU R 128 — the loudness recommendation that uses LUFS
• [7] Lipshitz, Wannamaker & Vanderkooy, “Quantization and Dither: A Theoretical Survey”, J. Audio Eng. Soc. 40 (1992), 355–375 — why undithered quantization error is signal-dependent

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• Evaluation
• VoiceCloning
• TTS
• QualityEngineering
• SageMaker