My Agent's Rulebook Was an Accidental Threat Model

Rules written after incidents

One of my agents produces narrated videos in my cloned voice. Some time ago the voice-cloning step failed, a fallback quietly switched to a stock text-to-speech voice, and the run reported success. The video had somebody else’s voice on it, and nothing in the run said so.

So I wrote a rule that a fallback may lower the quality of the output but never change who appears to be speaking. That rule is now in my setup, along with dozens of others like it. When I read through them recently I noticed that nearly every one had the same history. Something broke, and I wrote a rule that’s still there.

That’s a perfectly normal way for a system to grow. It’s also the opposite of what threat modeling tries to do, which Adam Shostack describes as the “measure twice, cut once” of security: find the problem while it is still cheap to fix [2]. So I took the rulebook and ran my whole setup through his Four Question Framework, to see what a planned version would look like next to the accidental one.

Shostack’s four questions

Shostack’s Four Question Framework is short enough to quote in full [1]:

1. What are we working on?
2. What can go wrong?
3. What are we going to do about it?
4. Did we do a good job?

It names the questions a threat model has to answer and leaves the technique to you, whether that’s a whiteboard sketch, a data flow diagram, STRIDE or a card game. The AWS Well-Architected Security Pillar names it as one way to start threat modeling a workload [6], and the Threat Modeling Manifesto is built on those four questions [5].

Shostack insists on the exact wording, and his 2024 whitepaper explains why each word was chosen [2]. People rephrase the questions all the time, and in his words “rephrasings often lose nuance, flexibility, or both.”

What are we working on?

My setup is a terminal agent (kiro-cli) with a few dozen skills, steering files and hooks, working over an Obsidian vault on my laptop. It talks to MCP servers for mail, calendar, CRM and chat, dispatches long-running jobs to a remote development machine and collects the results, and drives a browser for the sites that need my login. And it can commit to the repository behind this website, which then builds and deploys it. Sending the newsletter is a separate, explicit step.

Drawing it made the trust boundaries obvious. There are four trust zones: my laptop, the work cloud account, the account that hosts my personal site, and the public internet, and five arrows cross between them. Tool calls carry my mail, calendar, CRM and chat data with my permissions. Dispatch sends job prompts to the remote machine and brings the results back. A commit carries the authority to build and publish my site. The publish pipeline sends posts and emails to readers. And one arrow brings pages and mail from the public internet into the agent. The orange arrow is the one that matters most for an agent. The pages it fetches and the mail it reads come from people I don’t control.

Two words in the question changed how I answered it. The first is “we”. Shostack moved from “you” to “we” partly to make the perspective explicit, because “all threat modeling is done from a perspective” [2]. Mine is the operator of my own agent. The model provider, the mail service and the cloud services each have their own threat models. Their failures stay in my model as dependencies, but the mitigations on their side are theirs. Well-Architected adds, under Q3, that asking what you’re going to do about a threat also means asking who is responsible for doing something about it [6]. For me that’s my own configuration and scripts, and how I use them. A personal setup usually has one person doing the modeling, which the Threat Modeling Manifesto lists as an anti-pattern, the “Hero Threat Modeler” [5]. For a personal setup that’s hard to avoid, but on a team’s shared agent the “we” should be the people who use it.

The second is “working on”. The README notes that it replaced “building” [1], and the whitepaper explains that “building” pulls toward analyzing the whole system, including parts that already exist [2]. My setup has changed a lot over time, and if I tried to model every past version I’d never finish. So I modeled what it does today and ignored the history, except where the history explains a rule.

What can go wrong?

The whitepaper keeps this question in plain English on purpose, so that nobody needs to know STRIDE to take part [2]. So I wrote down what had gone wrong in my own words first, and only afterwards swept the list with two mnemonics: STRIDE for the whole system [15] and PHANTOM-B for the parts where an LLM makes decisions.

PHANTOM-B is Shostack’s 2026 analog to STRIDE for LLMs [3]. The letters stand for Prompt injection, Hallucination, Anthropomorphization, Non-explainability, Training issues, Over-reliance, Missing security engineering and Biases. He wants them used as prompts rather than categories, and says you “may discover a specific threat and have trouble putting it into a cubbyhole.” Some of mine did.

Here is what has actually gone wrong in my setup, as patterns rather than full incident reports.

The voice swap from the opening is the first. The fallback produced a different speaker and called it success, and the pipeline accepted the result because the run said it worked.

Another was two sessions on one working tree. I often run an interactive session and a background run at the same time, and one of them tidied up the git working tree and swept away uncommitted files the other was still working on.

Publishing to the wrong place also happened. Some teasers for scheduled posts carried a link to my local preview server where the public URL should have been. And a deploy script I had for publishing would also send the newsletter to every subscriber, as a side effect. Both change what readers receive, and the second one can’t be undone.

A cleanup filter matched too much. A teardown script picked what to delete by a substring in the resource name. Resources from another project had the same substring in their names, so the cleanup reached beyond the project it was written for.

An unauthenticated run reported zero. A monitoring job ran without credentials, got empty answers, reported zero, and saved that zero as the baseline for future comparisons. The report looked fine, but the saved baseline would have made the next authenticated run report every real item as new.

And a review on a different model flagged a passage that a same-model self-review had passed. A self-review on the same model that wrote a technical post confirmed a passage was correct. A review on a different model flagged the same passage as imprecise.

Sweeping the list with STRIDE and PHANTOM-B then added threats that hadn’t happened yet, or where I’d been lucky. Secrets can end up in session transcripts, which are stored and re-read later. Pages and emails the agent reads can carry instructions. And drafts can contain claims the model produced with confidence and no source.

PHANTOM-B’s prompt for prompt injection is “How could someone prompt inject?” [3]. For an agent that reads my mail, the honest answer is: by sending me an email. I described the wider version of this problem, agents whose input becomes their instructions, in The Agent Security Stack Nobody Is Building [13]. The PHANTOM-B paper calls prompt injection “currently unsolvable” because user prompts, system prompts and everything else get compressed into the same statistical tokens [3], so my answers are to limit what injected text can make the agent do, to detect it, and to accept that some risk remains.

What are we going to do about it?

Shostack’s wording here is deliberate too. It avoids “how will we fix it?”, because mitigating is only one of the options. You can also eliminate the risk, accept it, transfer it, or decide that something isn’t worth doing at all [2]. And it’s in the future tense, because a threat model produces a backlog and the fixing happens later as ordinary engineering work.

The most useful tool for this question came from another Shostack paper. In Inherent Threats he places every threat on a spectrum from accidental, through trade-off, to inherent [4]. Accidental threats exist because nobody thought to look, and they can be removed or fully mitigated. Inherent threats come with what the system is for. For those, he writes, “protective measures cannot be perfect or complete,” so detection and response take a larger share of the effort [4]. His own example from that paper: large language models “will hallucinate and show biases.”

Nearly everything that had actually happened to me landed in the accidental column, where each row has a mechanical fix.

I can’t eliminate the threats in the two inherent rows at the bottom. I mitigate, detect, contain and respond to them, and accept the residual risk. For hallucination that means a verification gate that catches claims before they ship. For prompt injection, containment means the agent can read a hostile email but shouldn’t send, delete or deploy anything on its own as a result. That rule is mostly steering text, not a hard boundary. In an interactive session the tools that send mail aren’t pre-approved, so the agent has to ask before it uses them. Shell commands and chat posts are pre-approved, though, so a delete, a deploy or a chat message can go through without a prompt. Background runs start with every tool trusted, so nothing asks there at all. Two caveats belong next to that row. The publish path is also an output channel, so text from a hostile page could reach a post if I stopped reading drafts before they ship. And a confirmation prompt moves the over-reliance to me. If I approve every request without reading it, the control is gone, so those prompts have to stay rare enough that each one still gets read.

Some rows rely on a script, hook or gate that runs every time and fails the run when the check fails, and others started as a sentence in a steering file asking the agent to behave. I wrote about this split in Security Is Job Zero: “Steering is guidance a probabilistic agent usually follows; the hooks and gates are the controls you can rely on” [11]. PHANTOM-B files this under anthropomorphization. “We also expect that adding ‘Do not do X’ will make it less likely to do X, because that’s how a human would behave. But perhaps we’re just adding some weight to whatever X is and making it more likely?” [3]

“What we’re not going to do” is also on Shostack’s list of valid outcomes [2]. My clearest example is the learning loop that reviews my sessions and proposes changes to skills and steering files. It doesn’t edit itself. As I wrote when I described that loop, “It proposes; I approve” [12]. I could let it apply its own suggestions, and some days that would save time. I decided against it, because a wrong suggestion applied automatically becomes a standing instruction nobody checked.

Two AWS posts on the same split

Two AWS Security Blog posts from early October 2026 come at the same split from a different direction [8] [9]. Nidhi Ramakant, Ievgeniia Ieromenko and Justin Kontny built what they call an AI vulnerability harness, a system that uses AI to triage vulnerability-scanner findings, and they ship its methodology as a Kiro steering file [10]. Their summary of what made it work: “The fix wasn’t a better model. It was better steering.” [9]

Their posts line up with the four questions in places I found useful. The steering file starts with structural checks: do the referenced files and functions exist, and does the data flow back up the claim? The authors report that in their testing “approximately 30% of unsteered model findings referenced code structures that didn’t exist,” and that the verification instruction produced zero fabricated paths [9], which is a measured Q3 answer to PHANTOM-B’s hallucination prompt. Their scoring has a floor of 0.15, because “no combination of controls reduces risk to zero” [9], which is the inherent-threats idea written as a formula. And their rule “Do not file P3 findings as security issues” [9] is a decision about what they’re not going to do, the outcome Shostack also lists.

They are also precise about the limits: “A steering file is methodology, not machinery” [9]. It tells the model how to think about findings, and it can’t do deterministic taint tracking or check live infrastructure. Part 1 puts this into the architecture, verifying AI-generated claims against the code before a human sees them [8]. A threat model tells you where the methodology needs machinery behind it. Their triage system is itself an LLM system, so it could get its own four-question pass.

Did we do a good job?

The fourth question started as a mechanical check (do we have a diagram, did we file tickets) and grew into reflection [2]. My setup has had a working answer to it for a while, without the name. After each session a background analyzer reads the transcript, looks for mistakes and corrections, and writes suggested changes for me to review [12]. Every retro section in my steering files is one of those answers.

Running the formal exercise showed me where that loop falls short. A few rules kept coming back, even though they were written down clearly. Rewording them didn’t change much. What helped was taking them out of prose: for background runs the launcher now appends the most important rules to every prompt, so no run starts without seeing them, though the agent still has to follow them. That’s the anthropomorphization problem from PHANTOM-B showing up in my own retro data.

The whitepaper offers two simple probes for this question: would you threat model again, and would you recommend it to colleagues [2]? For this exercise, most of the value came from Q1, because drawing the diagram showed me the untrusted-input arrow more clearly than any rule had. The rest came from the inherent column, because it told me which threats to stop trying to fix with more steering text.

The Manifesto words this question as “Did we do a good enough job?” [5]. The whitepaper keeps the shorter form because “good enough” can weaken the long-term aim [2]. For a personal setup I’m comfortable with “good enough for now,” as long as I ask again when the setup changes.

A template you can copy

This is the page I’d hand to anyone running an agent with access to their mail, accounts or publishing. Fill it in from your own perspective as the operator.

WHAT ARE WE WORKING ON?            (the setup as it is today)
- Draw it: agent, tools/MCP servers, accounts, anything it can publish to
- Mark trust boundaries; circle every arrow that carries untrusted input
- Perspective: "threats in scope for me as the operator of this agent"

WHAT CAN GO WRONG?                 (plain words first, then sweep)
- List what already went wrong, in your own words
- STRIDE per element: Spoofing, Tampering, Repudiation,
  Information disclosure, Denial of service, Elevation of privilege
- PHANTOM-B for each LLM step: Prompt injection, Hallucination,
  Anthropomorphization, Non-explainability, Training issues,
  Over-reliance, Missing security engineering, Biases

WHAT ARE WE GOING TO DO ABOUT IT?  (one row per threat)
| threat | mitigate / eliminate / accept / transfer / not doing |
| accidental / trade-off / inherent | control: check, hook or gate? or only text? |
- Inherent rows: how will I detect it, and what is the blast radius?

DID WE DO A GOOD JOB?
- Which rules keep being broken? Move them from text into a check
- Would I do this again? Would I recommend it?
- Revisit when the setup changes

If you’re running this on AWS

For an agent workload on AWS, the Well-Architected best practice SEC01-BP07 walks through the same four questions, suggests STRIDE for brainstorming, and asks you to scope Q3 to the mitigations under your control [6]. It points to Threat Composer, a tool from AWS Labs for writing threat statements [7]. Threat Composer uses a threat grammar that makes each Q2 answer specific: “A [threat source] with [prerequisites] can [threat action] which leads to [threat impact], negatively impacting [impacted assets].” [14] My voice-swap row would read: a failing synthesis step with a configured fallback can replace the cloned voice with a stock voice, which leads to a published video with the wrong narrator, negatively impacting the trust of the people who watch it.

If your agent uses AI to triage security findings, the two AWS posts and the sample repository are a ready-made starting point for the steering part [8] [9] [10].

What’s in your rulebook?

My guess is that most agent setups grow the way mine did, with a new rule after each incident. Four questions and an afternoon can show you which of those rules are controls and which are only hopes, and which threats can be removed and which residual threats need mitigation, detection, containment and response.

Which rule in your agent setup has been broken more than once, even though it was written down?

Sources

• [1] Shostack’s Four Question Frame for Threat Modeling, GitHub README (Adam Shostack, CC-BY)
• [2] Understanding the Four-Question Framework for Threat Modeling, White Paper #5 (Adam Shostack, November 2024)
• [3] PHANTOM-B: A STRIDE Analog for LLMs, White Paper #6 (Adam Shostack, July 2026)
• [4] Inherent Threats, White Paper #4 (Adam Shostack, March 2024)
• [5] Threat Modeling Manifesto
• [6] AWS Well-Architected Framework, SEC01-BP07 Identify threats and prioritize mitigations using a threat model
• [7] Threat Composer, AWS Labs on GitHub
• [8] Building your AI vulnerability harness, Part 1 (Nidhi Ramakant, Ievgeniia Ieromenko and Justin Kontny, AWS Security Blog, October 2026)
• [9] Configuring your AI vulnerability harness, Part 2: The steering file (Justin Kontny, Ievgeniia Ieromenko and Nidhi Ramakant, AWS Security Blog, October 2026)
• [10] sample-ai-vulnerability-harness-steering-file, AWS Samples on GitHub
• [11] Security Is Job Zero, Even (Especially) in the Age of Coding Agents (my earlier post)
• [12] How My Setup Learns While You Sleep (my earlier post)
• [13] The Agent Security Stack Nobody Is Building (my earlier post)
• [14] Threat modeling your generative AI workload to evaluate security risk (AWS Security Blog, source of the threat grammar wording)
• [15] The threats to our products (Loren Kohnfelder and Praerit Garg, April 1999, the paper that introduced STRIDE)