Vector Search, From the Whiteboard to the Cloud

🎬 Also available as a blog walkthrough video: a narrated screencast of this post.

A few months ago I argued that RAG is still needed, even with million-token context windows [1]. That post was about when to retrieve. This one goes one level down: what is the retrieval actually doing?

It is a fair question to keep asking, because the answer is deceptively simple. Vector search is one of those ideas you can sketch on a whiteboard in five minutes and then spend a quarter getting right in production. I want to do both halves here: the whiteboard intuition first, then what it takes to run it as a managed store on AWS. That pairing is the point of this new “Whiteboard to Cloud” format, and this is its first entry.

The whiteboard half: embeddings and nearest neighbours

The clearest short explanation I have seen is Mike Pound’s Computerphile video on vector search [2]. The intuition goes like this.

You take a piece of text and run it through an embedding model. The model does not predict the next word. It outputs a fixed list of numbers, say 500 of them, that places the text as a single point in a 500-dimensional space. Pound’s framing is that this is the same trick as Face ID: a network trained to put similar faces near each other in an embedded space, except here it does it for sentences.

The model learns this through contrastive training: show it many examples of text that means the same thing and many examples that does not, and teach it to place the similar ones close together. So “Why is the sky blue?” and “The sky is blue because of Rayleigh scattering” land near each other, while “Bicycles have two wheels” lands far away. In the video, the cosine distance between the two sky sentences is 0.2; between the sky and the bicycle sentence it is 0.94. Close to as different as you can get.

That word cosine matters. To measure similarity you do not compare the raw distance between two points. You compare the angle between the two vectors. Length is treated as noise; only direction carries meaning. Two sentences point in nearly the same direction when they mean nearly the same thing, regardless of how “loud” the embedding came out.

Two properties fall out of this that explain why it works so well in practice. First, you embed your corpus once, store the vectors, and then every query is a fast lookup against that index. Second, the matching is forgiving. Misspell “blue”, ask in slightly different words, or phrase the question awkwardly, and the embedding still lands close to the right neighbourhood. As Pound puts it, you never have to get it exactly right. For a retrieval system facing real users, that tolerance is a feature, not a bug.

So the whiteboard recipe is: embed the query, find the nearest stored vectors by angle, fetch the original text behind those vectors, and hand it to the model as context. Embed, index, search, retrieve. That is the whole idea.

Where it gets subtly wrong in production

The recipe is correct. It is also where most of the real work hides, because every step has a knob that changes the answer.

Chunking

You do not embed whole documents. You split them into chunks first, often a few hundred tokens with a small overlap so an important fact is not cut in half. Pound chunks a 170-page key-management PDF into roughly 800-token pieces with overlap before embedding. Chunk too large and a single vector tries to represent five unrelated ideas, so its direction means nothing in particular. Chunk too small and you shred the context a passage needed to make sense. There is no universal right answer; it depends on your documents, and it is the first thing I tune when retrieval quality is poor.

Recall versus latency

“Find the nearest vectors” is exact in the whiteboard version. At scale it is approximate. Approximate nearest neighbour (ANN) indexes trade a little accuracy for a lot of speed, and the knobs that make them fast can quietly drop the one chunk you needed. Asking for the top 10 instead of the top 3 raises recall but adds noise to the context. This is a tuning axis, not a default.

Embedding drift

The vectors are only comparable if they all came from the same model. Change the embedding model and every stored vector is now in a different space than your queries, so similarity becomes meaningless. Re-embedding the whole corpus is the price of an upgrade, and forgetting that is a classic production incident.

Metadata filtering

Pure vector similarity has no notion of “only this customer’s documents” or “only the current version.” In production you almost always want to filter by metadata first and search vectors second. Skipping this is how a retrieval system returns a confidently similar passage from the wrong tenant.

None of these are exotic. They are the difference between a demo that works on three sentences and a system that holds up against a real corpus.

The cloud half: a managed vector store on AWS

Here is the honest tension. Everything above is buildable yourself, and for a small corpus it is genuinely a weekend project: pick an embedding model, pick a vector library, write the chunking, store the vectors. But each of those knobs is also a thing to operate, monitor, and keep in sync as documents change.

On AWS, Amazon Bedrock Knowledge Bases runs that loop for you. Per AWS Prescriptive Guidance, after you point it at your data, a knowledge base “internally fetches the documents, chunks them into blocks of text, converts the text to embeddings, and then stores the embeddings in your choice of vector database,” and it keeps those embeddings in sync as the source data changes [3]. The four whiteboard steps, ingestion through storage, become managed.

It connects to data sources like Amazon S3, SharePoint, Confluence, Salesforce, and a web crawler, and it writes the vectors into a store of your choice. The supported list includes Amazon OpenSearch Serverless, Amazon Aurora PostgreSQL-Compatible (pgvector), Pinecone, Redis Enterprise, and MongoDB Atlas [3]. That set keeps growing as Bedrock adds stores, with more recent options such as Amazon S3 Vectors and Neptune Analytics. If you would rather not decide, Bedrock can create an OpenSearch Serverless store for you; if you have an existing database, you can bring it.

On the read side there are two APIs, and the distinction maps cleanly onto the two halves of this post [3]:

• Retrieve gives you just the retrieval half. It embeds your query, searches the knowledge base, and returns the matching chunks with scores. You decide what to do with them. This is the API to reach for when you want the nearest-neighbour result and nothing else.
• RetrieveAndGenerate does retrieval plus generation in one call, augmenting the prompt with the retrieved chunks and returning a model answer with traceable sources.

This series is about retrieval, so Retrieve is the one that matters here. A minimal call looks like this:

import boto3

client = boto3.client("bedrock-agent-runtime")

resp = client.retrieve(
    knowledgeBaseId="KB12345678",
    retrievalQuery={"text": "Why is the sky blue?"},
    retrievalConfiguration={
        "vectorSearchConfiguration": {"numberOfResults": 10}
    },
)

for r in resp["retrievalResults"]:
    print(r["score"], r["content"]["text"][:120])

That numberOfResults is the same recall-versus-noise knob from the whiteboard, now a single parameter. The chunking strategy, the embedding model, and the sync schedule are configuration on the knowledge base rather than code you maintain.

When I reach for managed, and when I do not

I want to be clear about the trade, because “managed” is not automatically the answer and “build it yourself” is not automatically purer.

My own setup is the counter-example. The assistant I use for this work is built on kiro-cli plus an Obsidian vault, and its retrieval over my notes is plain text search, not a vector store at all. For a few hundred markdown files that I wrote and whose vocabulary I know, keyword search is faster to reason about, has no embedding model to keep in sync, and never returns a confidently wrong neighbour. A managed knowledge base would be more machinery than the problem needs.

That is the actual decision criterion, and it has nothing to do with the cloud being good or bad. It is about corpus size, query volume, and how much of the operational loop you want to own.

Managed wins when the corpus is large or changing, when queries are unpredictable, and when keeping embeddings in sync with shifting source data is real ongoing work you would rather not own. That is most enterprise retrieval. DIY wins when the corpus is small, stable, and yours, and when the simplest thing that works is the right thing to ship. Both are legitimate. The mistake is reaching for the heavier option by reflex, or the lighter one out of stubbornness.

If you are running this on AWS

For a managed retrieval layer, the concrete mapping is:

• Amazon Bedrock Knowledge Bases for the managed ingest-chunk-embed-store loop and the Retrieve / RetrieveAndGenerate APIs [3].
• Vector store: start with Amazon OpenSearch Serverless (Bedrock can provision it for you), or bring Aurora PostgreSQL with pgvector if you already run Aurora and want vectors next to relational data [3].
• Data source: Amazon S3 is the simplest starting point; the same knowledge base also supports SharePoint, Confluence, Salesforce, and a web crawler, with incremental sync so only changed content is re-crawled [3].

The Retrieve snippet above is the whole read path. Everything else, the chunking strategy and the embedding model, is configuration on the knowledge base.

What is next in the series

This was the retrieval half. Part 2 will look at chunking and the embedding choices in more depth, where the production knobs above actually get set. Part 3 will connect retrieval to the rest of a working system, including how a model decides which tool or store to call, which is the thread I picked up in my MCP Strategies on AWS series [4]. The theme running through all three is the one from this post: the whiteboard model is correct, and the engineering is in the details it leaves out.

If you only keep one thing: vector search is “embed, then find the nearest neighbours by angle.” Everything in production is in service of making those neighbours the right ones.

💬 When did you last reach for a managed retrieval layer, and when did plain search turn out to be enough? I am curious where others draw that line.

Sources

• [1] My earlier post, “Is RAG Still Needed with 1M+ Token Context Windows?”: https://schristoph.online/blog/is-rag-still-needed/

• [2] Computerphile, “Vector Search with LLMs” (Mike Pound): https://www.youtube.com/watch?v=YDdKiQNw80c

• [3] AWS Prescriptive Guidance, “Knowledge bases for Amazon Bedrock” (managed RAG, vector stores, Retrieve / RetrieveAndGenerate): https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-fully-managed-bedrock.html

• [4] My series on tool and retrieval design in code, “MCP Strategies on AWS”: https://schristoph.online/blog/mcp-strategies-on-aws-overview/

#AI #RAG #VectorSearch #AmazonBedrock #Architecture

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AI
• RAG
• VectorSearch
• AmazonBedrock
• Architecture