From Prompt to Context to Memory: The Arc, Grounded on AWS

A while back I argued that context engineering had quietly replaced prompt engineering as the skill that decides whether an agent is any good [1]. That piece was the framing. This one is the build.

I want to walk the whole arc, from phrasing a prompt, to engineering the entire context window, to persisting memory across sessions, and show which managed AWS service owns each rung. There is runnable code for the load-bearing steps, in a public repo, so this is a map you can clone and start from rather than a diagram to nod at.

From prompt to context to memory

Prompt engineering was the first wave. You phrased the question carefully, added a few examples, pinned the output format, and iterated on wording until a single interaction behaved. It was a real skill, and for a single-purpose bot it can still be all you need.

The ceiling shows up when agents run across many steps, tools, and sessions. Output quality stops being dominated by how you ask and starts being dominated by what the model can see when it answers. IBM’s Martin Keen puts it plainly: model intelligence is often no longer the bottleneck; getting the right context to the model at the right time is an infrastructure problem [2].

The word “prompt” is overloaded, which hides the shift. Colloquially a prompt is what you type for one interaction. At the API level the same word covers the entire assembled input: instructions, tool definitions, history, retrieved passages. Context engineering is the system that assembles that whole input before the prompt ever arrives. Naming it matters, because it moves the work from wordsmithing a string to engineering a pipeline, the same move DevOps made when it dragged infrastructure from hand-edited config to versioned, tested, reviewed code. That parallel is not an accident: Patrick Debois, who coined “DevOps” in 2009, is the one who named this discipline “Context Is the New Code” [3].

The rungs stack. Here is the arc I will spend the rest of the post building on AWS.

Prompt engineering did not disappear. It became one input to a much larger system.

Context engineering is the whole window, not the prompt string

The definition I work from: context engineering designs systems that assemble the right information, in the right form, at the right time, into the model’s context window, treating the entire window as the unit of engineering.

A modern context window is roughly six components, and each carries its own discipline.

Prompt engineering addresses component one. Context engineering addresses all six. The cleanest one-liner I have found comes from Arize’s Sally-Ann Delucia: context decides what the model sees, memory decides what survives. Conflate the two and you get agents that know everything and understand nothing.

Best practices that actually move the needle

Retrieval discipline

Precision beats recall. Focused, relevant context outperforms a large context with the answer buried in noise, and a topically related but wrong chunk hurts more than unrelated filler. Deduplication and re-ranking earn their keep. When semantic similarity alone misses relationships, GraphRAG navigates entities instead.

Compression

Better context is more precise context, not more of it. Distil conversation history into summaries; store distilled episodes, not full transcripts.

The memory stack

The taxonomy worth internalising is a ladder of persistence (the CoALA framing, summarised by IBM) [4].

The dominant failure is collapsing all of these onto the working rung, pasting history, instructions, and reference docs into one giant prompt, then wondering why the agent got slower and less reliable.

Context isolation

Give each sub-task its own clean context so one agent’s working memory does not pollute another’s. Progressive disclosure helps too: load only a skill’s name and description at startup, and the full instructions when it matches, which keeps the working budget free until you need it.

Prompt caching

Mark a stable prefix once and reuse it cheaply on every turn (the AWS specifics are below).

Structured context and vocabulary

Order matters. Put static content first and variable content last, which also maximises cache hits. Treat vocabulary as context: ambiguous terms get amplified into confidently wrong output because agents interpret words however they like and rarely ask for clarification.

Evaluate context quality

Adding an instruction does not just add behaviour, it changes behaviour, so ripple effects are the rule. Because output is non-deterministic, use an error budget across repeated runs rather than a single pass/fail gate, and schedule independent evals to catch silent drift.

Manage tool results

Tool outputs land back in the window and compound fast. Summarise or truncate large results instead of letting a raw API dump crowd out the signal.

The limits nobody puts on the slide

Long context is not memory

A bigger window (128K, 1M tokens) lets an agent see more at once. It does not solve persistent storage, retrieval of the right thing, or learning from the past. Dan Biderman frames it as RAM versus disk: fitting more in the window is not the same as remembering and improving [5].

Lost in the middle, and context rot

Chroma’s study across 18 models is the load-bearing evidence: models do not process context uniformly, and performance degrades as input grows even on trivial tasks [6]. Needle-in-a-haystack benchmarks hide this because pure lexical retrieval is the easiest long-context task. Add semantic matching or distractors and quality drops. In one result, roughly 300 tokens of relevant context beat about 113K tokens with the answer buried in noise.

Cost and latency

Every token in the window is processed and billed on every turn. Large context is recurring cost, not free scale.

Retrieval failure modes

Chunk too large and one vector smears five ideas; too small and you shred the meaning. Change the embedding model and every stored vector is stale. Approximate-search knobs can silently drop the chunk you needed. And a metadata filter is a relevance tool, not a security boundary: derive the searchable corpus from the caller’s identity, do not lean on a query parameter for tenant isolation.

RAG is not memory

RAG is an access mechanism over a store. It retrieves; it does not consolidate experience or build intuition. The bridge from episodic to procedural, an agent that reviews what it did, extracts the pattern, and writes it back as a validated reusable skill, is an architecture you build, not a model feature you wait for.

How the stack maps to AWS

Here is the part the framing post could not give you. Each rung has a managed home on AWS. These compose into one pipeline but stay separate services, each with its own configuration, and none is on by default.

Prompt caching: reuse the stable prefix

Prompt caching is an optional Bedrock feature on supported models that cuts response latency and input token cost by caching a static prompt prefix [7]. You mark cache checkpoints on a contiguous prefix that stays stable between requests; editing it causes a miss. The detail people get wrong is the per-model minimum per checkpoint: Claude 3.7 Sonnet needs at least 1,024 tokens, while Claude Opus 4.5, Opus 4.6, Sonnet 4.5, and Haiku 4.5 need 4,096. There is a maximum of four checkpoints (Claude), a TTL that many models set to five minutes and that resets on each hit, and one constraint worth remembering: it is on-demand only, not batch. Order static content first so it caches, variable content last.

# Bedrock Converse with a cache checkpoint on the static prefix
system = [
    {"text": build_static_prefix()},        # large, STABLE: instructions + reference docs
    {"cachePoint": {"type": "default"}},     # cache everything above
]
messages = [{"role": "user", "content": [{"text": question}]}]  # variable, last

resp = client.converse(modelId=model_id, system=system, messages=messages)
print(resp.get("usage", {}))  # cacheReadInputTokens appears once caching is active

This is best practice made concrete: curate the context once, feed it cheaply every turn.

Knowledge Bases and embeddings: retrieval you control

Bedrock Knowledge Bases is the managed ingest-chunk-embed-store loop. Point it at a data source and it fetches documents, chunks them (default around 300 tokens, or fixed, semantic, hierarchical), embeds each chunk, and writes vectors plus a source mapping into your vector store, re-indexing incrementally on sync. It exposes two read APIs: Retrieve returns matching chunks and scores, and RetrieveAndGenerate folds retrieval and generation into one traceable call. For context-engineering control you usually want Retrieve, because then you decide what enters the window, and numberOfResults is the same recall-versus-noise knob from the whiteboard.

resp = client.retrieve(
    knowledgeBaseId=kb_id,                       # from env, never hardcoded
    retrievalQuery={"text": query},
    retrievalConfiguration={
        "vectorSearchConfiguration": {"numberOfResults": 8}  # fewer = less noise
    },
)

The embeddings underneath can be Amazon Titan Text Embeddings V2, which outputs 1,024, 512, or 256 dimensions and accepts up to 8,192 input tokens [8]. It is a single invoke_model call.

body = {"inputText": text, "dimensions": 1024, "normalize": True}
resp = client.invoke_model(modelId="amazon.titan-embed-text-v2:0", body=json.dumps(body))
vector = json.loads(resp["body"].read())["embedding"]   # 1024-dim vector

I ran that last call against Titan v2 and it returns a 1,024-dimension vector, which is what the sample in the repo asserts. The vector store itself stays a separate resource with its own scaling and billing, whether that is OpenSearch Serverless, Aurora with pgvector so vectors sit beside relational data, or a managed alternative. When you would rather not run the pipeline at all, Amazon Kendra offers permission-aware enterprise search with automatic connectors, trading pipeline control for lower operational load.

AgentCore Memory: the rung above the window

Bedrock AgentCore Memory is a fully managed service that addresses agent statelessness with two levels [9]. Short-term memory captures turn-by-turn interaction events within a session, stored instantly through CreateEvent. Long-term memory automatically extracts insights across sessions, such as user preferences, facts, and summaries, processed asynchronously by extraction strategies you configure. You do not write long-term records directly.

agentcore.create_event(
    memoryId=memory_id, actorId=actor_id, sessionId=session_id,
    payload=[
        {"conversational": {"role": "USER", "content": {"text": user_text}}},
        {"conversational": {"role": "ASSISTANT", "content": {"text": assistant_text}}},
    ],
)
# long-term extraction runs asynchronously; you retrieve consolidated records later

Short-term maps to working and episodic capture; the async strategies promote durable insight into long-term memory. The consolidation into validated reusable procedures, the episodic-to-procedural bridge, is still a pipeline you build on top. Retention and other limits evolve, so read the current AgentCore Memory documentation rather than a number I quote here.

Guardrails: governance at the boundary

Amazon Bedrock Guardrails screens both directions. On input it covers prompt attacks (jailbreak, prompt injection, prompt leakage) plus content, topic, and word policies. On output, contextual grounding checks flag answers that are not grounded in the retrieved context. It is one adversarial layer scoped to the checks you configure, not the whole defence, and injection carried inside retrieved documents needs controls around it too.

The reference architecture

The value of the table is that the services compose. Here is a single agent turn, in the order it actually flows. Every arrow is something you wire up.

Ingestion happens offline: on each sync the Knowledge Base chunks, embeds with Titan, and writes to the vector store. Per turn, the agent assembles the window static-first (so prompt caching hits), retrieves the relevant chunks, carries continuity from AgentCore Memory, and passes both directions through Guardrails. Retrieval precision, cache-friendly ordering, a real memory tier, and grounding checks are the four native moves that turn “we have a big window” into “we engineer the context.”

Anti-patterns, and what to measure

The failures are all traceable to the sections above. The context buffet, loading everything because the window is big, actively degrades output. Four rungs on one rung is the root of “it keeps forgetting.” A metadata filter used as a security boundary is not authorization. Distractor blindness optimises recall while similar-but-wrong chunks poison the answer. Silent staleness treats a rule written months ago as still true. And waiting for a memory feature treats consolidation as something a model release will hand you.

Measure context quality, not just model quality: retrieval precision at k and distractor rate, the grounding-check pass rate, cache hit rate and cached-token share, the age of your oldest load-bearing knowledge artifact, and an error budget across repeated runs rather than a single green run.

A worked example: my Obsidian-vault agent setup

I have run a personal, Obsidian-vault-backed agent setup for about a year, and it is the clearest first-party evidence I have that this stack is real work, not a diagram. It maps onto the memory ladder directly.

Working memory is the live session, the conversation and the files the assistant can see right now. Semantic memory is the vault itself, a couple of thousand linked notes of research and conventions, loaded as context when relevant. Retrieval over it is plain text search, not a vector store, and that is a deliberate choice: for a few-hundred-file corpus whose vocabulary I know and wrote, keyword search is faster to reason about, has no embedding to keep in sync, and fails legibly. That is the do-it-yourself side of the do-it-yourself-versus-managed call I wrote about separately [10]; a Knowledge Base earns its place when the corpus is large, multi-author, or changing under you.

Procedural memory is the set of skill files and steering rules I refine between sessions: when a workflow goes wrong once, I encode the fix as a rule, and they load on demand through progressive disclosure. Episodic memory is the weakest rung, session-handoff notes written at the end of a session plus a review step, and that honest gap is exactly the frontier the memory research points at. Every session starts as a new hire; the notes and rules are the institutional knowledge that stops me re-onboarding an employee every conversation.

Where this leaves you

Models and tools are commoditising. The accumulated, validated, governed context is the part that does not transfer, and on AWS it is buildable today with managed services, curated deliberately rather than dumped. Better context is precision, not volume, and precision is an engineering discipline with a concrete build map.

The code for the four load-bearing steps is on GitHub, sanitized and runnable [11]. Clone it, point it at your own account, and start on the rung you are missing.

What is your weakest rung right now: retrieval precision, or memory that survives the session? I would like to hear which one is costing you more.

Sources

• [1] Context Engineering: The Skill That Replaced Prompt Engineering — my earlier framing piece.
• [2] IBM Technology, “RAG, GraphRAG, and Context Engineering” — Martin Keen on context as the bottleneck: youtube.com/watch?v=pN-LfxNFiTc.
• [3] Patrick Debois, “Context Is the New Code” (AI Engineer Europe, 2026) — naming the discipline and its lifecycle: jedi.be.
• [4] IBM, “What Is AI Agent Memory?” summarising the CoALA taxonomy — arXiv:2309.02427.
• [5] Dan Biderman (Engram), on long context as RAM, not disk: youtube.com/watch?v=jhpmMTus5a0.
• [6] Chroma, “Context Rot: How Increasing Input Tokens Impacts LLM Performance” — trychroma.com/research/context-rot.
• [7] AWS, Prompt caching for faster model inference — docs.aws.amazon.com.
• [8] AWS, Amazon Titan Text Embeddings models — docs.aws.amazon.com.
• [9] AWS, Add memory to your Amazon Bedrock AgentCore agent — docs.aws.amazon.com.
• [10] Vector Search, From the Whiteboard to the Cloud — the managed RAG loop and the do-it-yourself-versus-managed decision.
• [11] Companion code on GitHub — runnable prompt caching, Knowledge Bases, AgentCore Memory, and Titan embedding samples.
• [12] The Agent Memory Spectrum — the memory ladder in depth.