Agent Memory Is a Spectrum, Not a Switch

🎬 Also available as a blog walkthrough video: a narrated screencast of this post.

Your agent doesn’t have a memory problem. It has a memory-architecture problem.

I hear the symptom in customer conversations almost every week: “The agent keeps forgetting what we told it.” The reflex is to reach for a bigger context window or a better model, as if memory were a dial you turn up. It isn’t. Memory is something you design, and the design space is wider than “remember the conversation.”

I have a personal stake in getting this right. My own daily workflow runs on kiro-cli with an Obsidian vault as the backbone, and that setup only works because different kinds of “remembering” live in different places. The vault holds what I know. The session holds what I’m working on right now. And the skills, steering, and hooks I refine between sessions hold how I work. Three different mechanisms, three different rungs on the same ladder. Once I saw my own setup that way, the customer question answered itself.

Most AI Has Amnesia

The computational neuroscientist Artem Kirsanov has a sharp way of framing the gap. In his video “Most AI has amnesia. Here’s the fix,” he points out that a classic feedforward network like AlexNet has “no concept of time” [1]. You can feed it the frames of a movie one at a time, and it will classify each frame perfectly while having no idea the previous frame ever existed. Shuffle the frames into random order and the network wouldn’t notice. As Kirsanov puts it, it behaves “like an expert with an extreme case of retrograde amnesia.”

His fix is architectural: recurrent networks that bake the idea of sequence into the model itself. The lesson generalizes well beyond RNNs. Memory is not a free property that falls out of a big enough model. You build it into the system on purpose, or you don’t have it.

This is the same shape as an argument I made earlier about reasoning. Reasoning is a spectrum, not a switch [2], and so is memory. The interesting question is never “does it have memory?” It’s “what kind, and where does it live?”

The Spectrum

Researchers categorize agent memory much the way psychologists categorize human memory. The most cited source is the CoALA paper [3], “Cognitive Architectures for Language Agents,” from a team at Princeton, which IBM’s overview summarizes well [4]. Four types come up again and again: working, semantic, procedural, and episodic.

I find it useful to lay them out as a ladder of increasing persistence, from an agent with no memory at all to one that consolidates experience into durable knowledge.

One honest caveat before we climb it. This ladder is a design aid, not a biological hierarchy. CoALA and IBM treat episodic, semantic, and procedural memory as three siblings under “long-term memory,” not as strictly sequential stages [4]. A coding agent might need procedural memory long before it needs a knowledge graph. So read the spectrum as a menu of persistence levels you can mix, not a staircase you must climb in order.

Here is each rung, with the mechanism that actually implements it.

No memory is exactly that: a stateless call. Useful, and often the right choice. A classification endpoint or a single-shot extraction task needs no memory at all, and adding it would only add cost and failure modes. Not every agent needs to remember.

Working memory is the context window: fast, immediately accessible, and volatile, like RAM. Everything the agent can “see” right now lives here, and when the session ends it’s gone. Every chatbot has this. It is table stakes, and it is also where most teams stop, which is the root of the “it keeps forgetting” complaint.

Episodic memory is a record of past interactions and what happened in them. This is what lets an agent say “last time we tried X, it failed.” Production systems rarely store full transcripts. They store distilled episodes, deciding what is worth keeping based on whether it will be useful later. Forgetting, for an agent, is an engineering decision rather than a flaw.

Procedural memory is how the agent knows how to do things. The emerging open standard here is the skill file: a lightweight index the agent scans, with full instructions loaded only when a matching task arrives. This progressive-disclosure pattern keeps the working-memory budget free until a skill is actually needed.

Semantic memory is the agent’s knowledge base: facts, rules, conventions, documentation. In production this is often plain markdown loaded at session start, and at the far end it becomes a vector store or a knowledge graph. This is the rung where retrieval lives, which is exactly where my earlier question about RAG lands [5]. RAG is one mechanism for semantic memory, not a synonym for memory itself.

Rung to Mechanism

The whole point of the spectrum is that each rung has a concrete implementation. If you can name the rung, you can pick the mechanism.

The failure mode I see most often is collapsing all four into the context window. Teams paste history, instructions, and reference docs into one giant prompt, then wonder why the agent gets slower, more expensive, and less reliable. That’s four kinds of memory crammed onto one rung. Pull them apart and each gets a mechanism suited to it.

Where My Own Setup Sits

This is not theory for me. My day-to-day runs on kiro-cli with an Obsidian vault, and it maps onto the spectrum cleanly:

• Working memory is the live session: the conversation and files the assistant can see right now.
• Semantic memory is the vault: hundreds of linked notes, research, and conventions that persist across every session and get loaded as context when relevant.
• Procedural memory is the set of kiro-cli skills, steering, and hooks I refine between sessions. When a workflow goes wrong once, I encode the fix as a skill or a steering rule, and the next session does it right. That refinement loop is procedural memory in the most literal sense: the system learning how to do a task and keeping that knowledge.
• Episodic memory is the weakest rung in my setup today: session handoffs and notes capture some of it, but most of what happened in a given session still fades.

That episodic gap is the interesting frontier, and it’s where the next two posts in this series go.

The Frontier: Consolidation

The rung that turns memory into something resembling learning is the bridge from episodic to procedural: an agent that reviews what it did, finds the patterns, and writes them back as reusable skills. Davide Gallitelli published a clear reference implementation of exactly this on Amazon Bedrock AgentCore, where worker agents emit episodic trajectories and a scheduled job consolidates them into validated, reusable skills [6]. It mirrors how human sleep consolidates the day: the raw episodes fade, the lessons stay.

I’m deliberately not deep-diving that architecture here. It’s the subject of Part 3. For now the point is narrower: consolidation is the mechanism that moves an agent up from “remembers what happened” to “got better because of it,” and it is an architecture you build, not a model feature you wait for.

If You’re Running This on AWS

The spectrum is vendor-neutral, but each rung has a natural home in the AWS toolkit, and these are capabilities to reach for, not boxes to check:

• Working memory is the model’s context window on Amazon Bedrock. Larger windows give you more room, but per the spectrum, that’s still one rung.
• Semantic memory maps to Amazon Bedrock Knowledge Bases, the managed retrieval-augmented generation layer, with Amazon Neptune when the knowledge is graph-shaped [7].
• Episodic and procedural memory are what Amazon Bedrock AgentCore Memory is built for: short-term memory within a session and long-term memory that persists and is extracted across sessions, so agents carry forward what they learned [8].

The managed services exist precisely so you don’t hand-roll a session store and a consolidation pipeline before you’ve validated the idea. That’s the rung-to-mechanism mapping, done for you.

Pick the Rung First

So the next time an agent “keeps forgetting,” resist the urge to turn up the context window. Ask which rung the missing capability lives on. Does it need to recall this session (working), what happened in past sessions (episodic), how to do a recurring task (procedural), or a body of facts (semantic)? Name the rung, then pick the mechanism. That single reframe turns a vague complaint into an architecture decision you can actually make.

This is Part 1 of a three-part series on agent memory. Part 2 looks at treating a vault as a knowledge graph for semantic memory, and Part 3 digs into episodic-to-procedural consolidation.

Where does your agent sit on the spectrum today, and which rung is it missing?

Sources

• [1] Artem Kirsanov — “Most AI has amnesia. Here’s the fix” — YouTube
• [2] What Reasoning Actually Means — schristoph.online
• [3] CoALA: Cognitive Architectures for Language Agents — Sumers, Yao, Narasimhan, Griffiths (Princeton)
• [4] What Is AI Agent Memory? — IBM Think
• [5] Is RAG Still Needed with 1M+ Token Context Windows? — schristoph.online
• [6] Learning from Experiences: Give Your AI Agents a Memory That Compounds — Davide Gallitelli
• [7] Amazon Bedrock Knowledge Bases — AWS Documentation
• [8] Amazon Bedrock AgentCore Memory — AWS Documentation

#AI #AgentMemory #AgenticAI #Architecture #AmazonBedrock

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AI
• AgentMemory
• Architecture
• AgenticAI
• AmazonBedrock