Architecting Skills: How Code Makes AI Agents More Reliable Over Time

The Skill That Worked, Sometimes

The first version of almost every skill I build is a markdown file. It reads like a runbook: “search my inbox for links I sent myself, summarise each one, append them to the reading list, then move the processed mails out of the inbox so they do not get picked up again.”

A capable model can follow that. The first time, it usually does. The fifth time, something slips. It summarises the same three articles it already processed last run, because the final instruction (move the mail out of the inbox) is one sentence buried at the end, and the model treated it as optional. Nothing crashed. The output just quietly drifted from what I wanted.

That is the signature failure of a prose-only skill. It does not error. It improvises. And improvisation is exactly what you want when the task is “write a thoughtful summary” and exactly what you do not want when the task is “mark this item as done.”

I run my whole Solutions Architect workday through a set of these skills on a terminal agent [5]. Over time a clear pattern emerged in how each one matures, and it is the same pattern every time.

The Pattern: Observe, Extract, Test, Repeat

A skill gets more reliable through a loop that looks like this:

1. Observe a failure. Not a crash, usually. A drift. The agent did the task in a slightly wrong way, or in a way that worked once and broke on the edge case.
2. Find the mechanical step underneath. Ask what part of the workflow had a single correct answer that the model got wrong. Moving a file. Computing a date. Paginating an API. Formatting a table.
3. Extract that step into code. Replace the sentence of prose with a script the agent calls. The script has no opinion. It does the same thing every time.
4. Test it and move on. The next failure you observe is somewhere else, because that class of failure is now gone for good.

The interesting part is what stays in prose. You never push “decide which of these articles is worth featuring” into a script, because that is a judgment call, and a script cannot make it. You push “move the processed mail to the archive folder” into code, because that is mechanical, and the model should never have been trusted with it in the first place.

Each turn of the loop shifts a little more of the skill from “the model interprets this” to “the machine guarantees this.” The skill does not get smarter. It gets narrower in the places where narrow is correct.

Three Skills, Three Migrations

This is easier to see in real examples. Here are three skills from my own setup and the migration each one went through.

The morning brief: from one prompt to a pipeline

The morning brief started as a single instruction: “sync my calendar, scan my channels, check for new research, and write me a brief.” It was one prose blob handed to one agent.

It worked on light days and fell apart on heavy ones. With a full calendar and a dozen busy channels, the agent ran out of working context halfway through and started skipping steps to finish, without telling me which steps it skipped. The brief looked complete. It was not.

The fix was not a better prompt. It was structure. The skill became a pipeline with a thin orchestrator that delegates the heavy, independent steps (channel scanning, video research) to subagents with their own tool access, then assembles the result. Two deterministic guarantees went in alongside the prose: a file-existence check so a second run on the same day exits instantly instead of regenerating, and a verification gate so each step reports success only when its deliverable actually exists, not when the agent believes it is done. In an end-to-end test the full pipeline ran in about seven and a half minutes with no skipped-but-claimed steps. The judgment (“what matters today”) stayed in prose. The orchestration (“never drop a step to save context”) became mechanical.

The reading list: one sentence that needed to be a step

The reading-list skill had the bug I opened with. The instruction to move processed mails out of the inbox lived in prose, and prose instructions at the end of a long workflow are the first thing a model drops.

Here is the whole bug. The instruction was the tail of a longer sentence, easy to gloss over:

Search the inbox, summarise each link, append them to the reading list,
then move the processed mails out so they are not picked up next time.

The fix was to stop hiding it inside another step and make it a step of its own, with the re-run rule stated out loud:

4. Archive processed mail — REQUIRED, never skip.
   Move every mail handled in step 3 to `processed/`.
   On re-runs, skip any mail already in `processed/`.

The migration was small and boring, which is the point. The “move the mail” step became an explicit, non-optional operation the skill performs, not a polite request at the end of a paragraph. Re-runs stopped reprocessing the same items. One class of drift, gone, permanently.

LinkedIn analytics: from improvisation to a fixed path

Collecting post analytics started as “open the analytics page and read the numbers.” The agent improvised its way through the UI differently each time, and the aggregate page it landed on returned intermittent server errors, so some runs came back with partial data and no clear signal that they were partial.

The reliable version stopped improvising. It walks a known set of per-post URLs in a fixed order, because those pages are stable where the aggregate one is not, and it does the data collection as its own focused step rather than interleaving it with other work that competes for the same browser session. The numbers stopped being lucky.

None of these three changes made the model more capable. Each one took a step the model kept getting subtly wrong and handed it to something that cannot get it wrong.

The Summit Argument, One Level Down

Reliability was one of the toughest production problems I walked through at AWS Summit Hamburg, in a talk about moving agents from demo to deployment. The room was full, the questions kept coming, and a marching band with dancers passed the breakout room mid-sentence, which I am told is a Hamburg speaking achievement.

The argument I made there scales down neatly to a single skill. An agent in production fails for the same reason a prose skill drifts: a non-deterministic component was placed where a deterministic one belonged. The model is a probabilistic engine. Point it at a creative task and that is a feature. Point it at “paginate this API and total the results” and the same probability that gives you good prose gives you a wrong sum every so often. I have argued the coding-tools version of this before: clean, deterministic foundations are what make agents dependable, and the codebase, not the model, is usually the limiter [4].

Code is the cure precisely because it is dumb. A script that totals a list returns the same total every time, with no opinion and no creativity, and that is exactly what you want from a totalling step. This is reproducibility, not infallibility: a script can still be wrong, but it is wrong the same way every run, which is what makes it debuggable. A model that is wrong differently each time is the harder thing to pin down. The skill of building reliable agents is mostly the skill of knowing which steps deserve a dumb, guaranteed answer and which deserve a smart, flexible one.

The Boundary That Shifts

Early skills are almost all prose. Mature skills are mostly code with prose reserved for the parts that genuinely need judgment. The boundary between them is not fixed. It moves toward code every time you observe a mechanical failure.

A useful way to picture a mature skill: a thin layer of prose for the decisions, sitting on a thick layer of code for the mechanics, with a clear line between them.

The mistake I made early on was treating a skill as one thing, either a prompt or a program. It is both, and the value is in the seam. Put a decision in code and it becomes rigid and wrong at the edges. Put a mechanical step in prose and it drifts. The work is in routing each step to the right side. It is the same trap as the popular “one tool versus another” debates, which argue about the wrong axis and miss the question that actually decides the outcome [6].

A Maturity Model for Skills

After enough of these migrations, the stages are predictable enough to name.

• Level 1, pure prose. A markdown runbook. Flexible, fast to write, unreliable under load. Every new skill starts here, and that is fine.
• Level 2, prose plus scripts. The two or three steps that kept failing are now code. The skill stops drifting on those specific things.
• Level 3, pipeline with gates. The skill has structure: verification that a step really finished, idempotency so re-runs are safe, and delegation so a big job does not exhaust one agent’s context. This is where a skill becomes something you trust unattended.
• Level 4, mostly code. Prose survives only where judgment lives. The skill reads less like a runbook and more like a small program with a few human decisions wired in.

Most of my skills sit at Level 2 or 3. They climb a level each time I catch them failing. None of them needed to start higher, and trying to build a Level 4 skill on day one is wasted effort, because you do not yet know which steps will actually fail.

A Level 2 skill looks almost exactly like its Level 1 prose, with one difference: the step that kept drifting now ends in a script call instead of a description. The judgment stays in words; the mechanics become a command.

## Compute the reporting window

Pick which weeks belong in this report. Holidays and launch dates move the
boundary, so this is a judgment call — make it in prose. Then hand the dates
to the script and let it do the counting, never count days yourself:

    python3 scripts/reporting-window.py --quarter Q2

The prose says what to decide. The script guarantees the arithmetic. That one seam is the whole of Level 2.

Should This Step Be Prose or Code?

When you are deciding where a step belongs, one question settles it: does this step have a single correct answer?

“Summarise this article well” has no single correct answer, so it stays prose. “Move this mail to the archive” has exactly one correct outcome, so it belongs in code. Most steps sort themselves the moment you ask the question honestly.

There is a second principle worth stating plainly, because it is the one people skip: if a step can be skipped, an agent will eventually skip it. Anything that must always happen, a security check, a cleanup step, a final move, cannot live as a hopeful sentence at the end of a paragraph. It has to be a step the skill performs, or a guarantee the code enforces.

The strongest version of that guarantee in my setup is a hook. Before the agent is allowed to run a tool, a small script gets to veto the call, and a non-zero exit makes the block non-negotiable:

#!/usr/bin/env bash
# PreToolUse hook: refuse any file write that targets a protected path.
# Exit 2 hard-blocks the tool call — the agent gets the error, not the write.
target="$(jq -r '.tool_input.path // empty')"

case "$target" in
  *.env | */secrets/* | */.ssh/*)
    echo "Blocked: $target is protected and must never be written here." >&2
    exit 2
    ;;
esac
exit 0

The model never has to remember the rule, because it cannot get past it. A database vendor documented a sharp version of this: without explicit guidance, an agent created a view over a protected table that bypassed row-level security and exposed data, and the same agent got it right once the critical rule was made an unmissable, opinionated instruction rather than a footnote. Reliability is partly about extracting code, and partly about refusing to let the important steps be optional.

If You’re Running This on AWS

My setup is personal and sandboxed, which is why prose-to-code migration is enough on its own. At enterprise scale you want the same prose-versus-code boundary, but with managed reliability primitives underneath instead of scripts you maintain by hand. On AWS, Amazon Bedrock AgentCore is built for exactly this split [1].

AgentCore is a set of composable services for running agents in production, and they map cleanly onto the layers from the diagram above [2]:

The shape is the same one a single skill teaches you. The model handles judgment. Deterministic, managed services handle the mechanics, the state, the verification, and the access. The boundary between them is where reliability is won or lost.

For authoring the prose-and-code split itself, the agent IDE I use day to day, Kiro, encodes the same idea: skills capture reusable workflows, steering files hold the persistent rules the agent must always follow, and on-demand tool loading keeps the mechanical capabilities available without flooding the context [3]. It is the maturity model expressed as a tool: write the judgment as prose, pin the rules as steering, push the mechanics into code.

What I’d Tell My Earlier Self

Do not try to write the perfect skill. Write the prose version, use it, and watch where it drifts. Every drift is a signpost pointing at a step that wanted to be code. Follow enough of those signposts and the skill becomes something you can run while you are not watching, which is the only definition of reliable that matters.

The model is not the bottleneck. The boundary is. Get the boundary right and a good-enough model becomes a dependable one.

Where is the line between prose and code in the agents you are building, and what was the last step you moved across it?

Sources

• [1] Amazon Bedrock AgentCore — aws.amazon.com
• [2] Amazon Bedrock AgentCore FAQs (composable capabilities: Runtime, Gateway, Memory, Identity, Observability, Code Interpreter, Evaluations, Policy controls) — aws.amazon.com
• [3] Kiro — Spec-Driven Agentic IDE — kiro.dev
• [4] Stefan Christoph, “Code Quality Is the New Infrastructure” — schristoph.online
• [5] Stefan Christoph, “The Coding Agent That Doesn’t Code” — schristoph.online
• [6] Stefan Christoph, “CLI vs MCP: The Wrong Debate” — schristoph.online

#AgenticAI #Skills #Reliability #SoftwareEngineering #Kiro

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AgenticAI
• Skills
• Reliability
• SoftwareEngineering
• Kiro