Vector Search, From the Whiteboard to the Cloud

🎬 Also available as a blog walkthrough video: a narrated screencast of this post.

A few months ago I argued that RAG is still needed, even with million-token context windows [1]. That post was about when to retrieve. This one goes one level down: what is the retrieval actually doing?

It is a fair question to keep asking, because the answer is deceptively simple. Vector search is one of those ideas you can sketch on a whiteboard in five minutes and then spend a quarter getting right in production. I want to do both halves here: the whiteboard intuition first, then what it takes to run it as a managed store on AWS. That pairing is the point of this new “Whiteboard to Cloud” format, and this is its first entry.

The whiteboard half: embeddings and nearest neighbours

The clearest short explanation I have seen is Mike Pound’s Computerphile video on vector search [2]. The intuition goes like this.

You take a piece of text and run it through an embedding model. The model does not predict the next word. It outputs a fixed list of numbers, say 500 of them, that places the text as a single point in a 500-dimensional space. Pound’s framing is that this is the same trick as Face ID: a network trained to put similar faces near each other in an embedded space, except here it does it for sentences.

Text-embedding models typically learn this through contrastive training: show the model many examples of text that means the same thing and many examples that does not, and teach it to place the similar ones close together. So “Why is the sky blue?” and “The sky is blue because of Rayleigh scattering” land near each other, while “Bicycles have two wheels” lands far away. In the video, the cosine distance between the two sky sentences is 0.2; between the sky and the bicycle sentence it is 0.94. Worth pinning down what that number means: with the usual definition of cosine distance as one minus cosine similarity, it runs from 0 for the same direction, through 1 for perpendicular, to 2 for exactly opposite. So 0.94 says the two vectors are very nearly perpendicular, unrelated, as far as this model is concerned. It is not near the arithmetic maximum, though in practice text embeddings rarely point in opposing directions, so perpendicular is about as unrelated as you tend to see.

That word cosine matters. Under cosine similarity you do not compare the raw distance between two points. You compare the angle between the two vectors. Length is treated as noise; only direction carries meaning. (That is a property of this metric, not of vector search in general, other setups score by dot product or Euclidean distance, where magnitude does matter, and the right choice is whatever the embedding model was trained for.) Two sentences point in nearly the same direction when they mean nearly the same thing, regardless of how “loud” the embedding came out.

Two properties fall out of this that explain why it works so well in practice. First, you embed your corpus once, store the vectors, and then every query is a lookup against that index rather than a fresh pass over the corpus, how fast, and at what accuracy cost, is a production question we get to below. Second, the matching is forgiving. Misspell “blue”, ask in slightly different words, or phrase the question awkwardly, and the embedding typically still lands close to the right neighbourhood, how forgiving depends on the embedding model and the language. As Pound puts it, you never have to get it exactly right. For a retrieval system facing real users, that tolerance is a feature, not a bug.

So the whiteboard recipe is: embed the query, find the nearest stored vectors by angle, fetch the original text behind those vectors, and hand it to the model as context. Embed, index, search, retrieve. That is the whole idea.

Where it gets subtly wrong in production

The recipe is correct. It is also where most of the real work hides, because every step has a knob that changes the answer.

Chunking

You usually do not embed whole documents, a short, single-topic document that fits the model can go in whole, but long or multi-topic ones get split into chunks first, often a few hundred tokens with a small overlap so an important fact is not cut in half. Pound chunks a 170-page key-management PDF into roughly 800-token pieces with overlap before embedding. Chunk too large and a single vector tries to represent five unrelated ideas, so its direction means nothing in particular. Chunk too small and you shred the context a passage needed to make sense. There is no universal right answer; it depends on your documents, and it is the first thing I tune when retrieval quality is poor.

Recall versus latency

“Find the nearest vectors” is exact in the whiteboard version. Exact search stays available at scale too, it is a linear scan, Θ(Nd) work for N stored vectors of d dimensions, and for plenty of corpora that is entirely fine. What pushes you off it is a latency or resource budget, not size as such. Approximate nearest neighbour (ANN) indexes trade recall for speed, how much of each depends on the index, the data, and its parameters, and the knobs that make them fast can quietly drop the one chunk you needed. Asking for the top 10 instead of the top 3 returns more candidates and so improves the odds the right chunk is among them, but it also adds noise to the context. This is a tuning axis, not a default.

Embedding drift

The vectors are only comparable if they live in the same embedding space, in practice: same model, same version, same dimensionality, same preprocessing, unless the provider explicitly documents compatibility. Change any of those and every stored vector is now in a different space than your queries, so similarity becomes meaningless. Re-embedding the whole corpus is the usual price of an upgrade, and forgetting that is a classic production incident.

Metadata filtering

Pure vector similarity has no notion of “only this customer’s documents” or “only the current version.” So you constrain the search with metadata, but be deliberate about what that constraint is load-bearing for, because two different jobs get conflated here.

Relevance scoping, only the current version, only this product line, is exactly what metadata filters are for. Tenant isolation is not. A metadata predicate is just a query parameter: anything that drops it, mis-templates it, or lets user input influence it hands back a confidently similar passage from the wrong tenant. Derive the searchable corpus from the caller’s authorization instead, separate indexes, collections, or knowledge bases per tenant, and treat metadata filtering as a further constraint inside that already-scoped set rather than as the boundary itself.

It is also worth knowing that you generally do not control when the filter is applied. Stores variously implement pre-filtering, post-filtering, or filtered traversal inside the ANN index, and the choice affects recall and cost as well as semantics. Check what your vector store actually documents rather than assuming the filter runs first.

None of these are exotic. They are the difference between a demo that works on three sentences and a system that holds up against a real corpus.

The cloud half: a managed retrieval layer on AWS

Here is the honest tension. Everything above is buildable yourself, and for a small corpus it is genuinely a weekend project: pick an embedding model, pick a vector library, write the chunking, store the vectors. But each of those knobs is also a thing to operate, monitor, and keep in sync as documents change.

On AWS, Amazon Bedrock Knowledge Bases runs that loop for you. Per AWS Prescriptive Guidance, after you point it at your data, a knowledge base “internally fetches the documents, chunks them into blocks of text, converts the text to embeddings, and then stores the embeddings in your choice of vector database,” and when you sync, an ingestion job you trigger, it re-indexes incrementally, processing only documents added, modified, or deleted since the last sync [5]. The four whiteboard steps, ingestion through storage, become managed.

It connects to data sources like Amazon S3, SharePoint, Confluence, Salesforce, and a web crawler, and it writes the vectors into a store of your choice. The supported list includes Amazon OpenSearch Serverless, Amazon Aurora PostgreSQL-Compatible (pgvector), Pinecone, Redis Enterprise, and MongoDB Atlas [3]. That set keeps growing as Bedrock adds stores, with more recent options such as Amazon S3 Vectors [6] and Neptune Analytics [7]. If you would rather not decide, Bedrock can create an OpenSearch Serverless store for you; if you have an existing database, you can bring it. Either way, the vector store remains a separate resource with its own scaling, security, and billing, Knowledge Bases orchestrates the loop around it rather than replacing it.

On the read side there are two APIs, and the distinction maps cleanly onto the two halves of this post [3]:

• Retrieve gives you just the retrieval half. It embeds your query, searches the knowledge base, and returns the matching chunks with scores. You decide what to do with them. This is the API to reach for when you want the retrieval results and nothing else.
• RetrieveAndGenerate does retrieval plus generation in one call, augmenting the prompt with the retrieved chunks and returning a model answer with traceable sources.

This series is about retrieval, so Retrieve is the one that matters here. A minimal call looks like this:

import boto3

client = boto3.client("bedrock-agent-runtime")

resp = client.retrieve(
    knowledgeBaseId="KB12345678",
    retrievalQuery={"text": "Why is the sky blue?"},
    retrievalConfiguration={
        "vectorSearchConfiguration": {"numberOfResults": 10}
    },
)

for r in resp["retrievalResults"]:
    print(r["score"], r["content"]["text"][:120])

That numberOfResults is the same recall-versus-noise knob from the whiteboard, now a single parameter. The chunking strategy and the embedding model are configuration on the knowledge base rather than code you maintain; syncing is an ingestion job you trigger, on whatever schedule you choose to automate [5].

When I reach for managed, and when I do not

I want to be clear about the trade, because “managed” is not automatically the answer and “build it yourself” is not automatically purer.

My own setup is the counter-example. The assistant I use for this work is built on kiro-cli plus an Obsidian vault, and its retrieval over my notes is plain text search, not a vector store at all. For a few hundred markdown files that I wrote and whose vocabulary I know, keyword search is faster to reason about, has no embedding model to keep in sync, and fails legibly, it can still miss or return the wrong file, but never a confidently wrong neighbour whose relevance I cannot trace back to a query term. A managed knowledge base would be more machinery than the problem needs.

That is the actual decision criterion, and it has nothing to do with the cloud being good or bad. It is about corpus size, query volume, and how much of the operational loop you want to own.

Managed wins when the corpus is large or changing, when queries are unpredictable, and when keeping embeddings in sync with shifting source data is real ongoing work you would rather not own. That covers much of the enterprise retrieval I see. DIY wins when the corpus is small, stable, and yours, and when the simplest thing that works is the right thing to ship. Both are legitimate. The mistake is reaching for the heavier option by reflex, or the lighter one out of stubbornness.

If you are running this on AWS

For a managed retrieval layer, the concrete mapping is:

• Amazon Bedrock Knowledge Bases for the managed ingest-chunk-embed-store loop and the Retrieve / RetrieveAndGenerate APIs [3].
• Vector store: start with Amazon OpenSearch Serverless (Bedrock can provision it for you), or bring Aurora PostgreSQL with pgvector if you already run Aurora and want vectors next to relational data [3].
• Data source: Amazon S3 is the simplest starting point; the same knowledge base also supports SharePoint, Confluence, Salesforce, and a web crawler, with incremental sync so only content that changed since the last sync is re-processed [5].

The Retrieve snippet above is the minimal happy-path read call. The chunking strategy and the embedding model are configuration on the knowledge base; IAM permissions, filtering, retries, quotas, and monitoring are still yours to handle around it.

What is next in the series

This was the retrieval half. Part 2 will look at chunking and the embedding choices in more depth, where the production knobs above actually get set. Part 3 will connect retrieval to the rest of a working system, including how a model decides which tool or store to call, which is the thread I picked up in my MCP Strategies on AWS series [4]. The theme running through all three is the one from this post: the whiteboard model is correct, and the engineering is in the details it leaves out.

If you only keep one thing: vector search is “embed, then find the nearest neighbours by angle.” Everything in production is in service of making those neighbours the right ones.

💬 When did you last reach for a managed retrieval layer, and when did plain search turn out to be enough? I am curious where others draw that line.

Sources

• [1] My earlier post, “Is RAG Still Needed with 1M+ Token Context Windows?”: https://schristoph.online/blog/is-rag-still-needed/

• [2] Computerphile, “Vector Search with LLMs” (Mike Pound): https://www.youtube.com/watch?v=YDdKiQNw80c

• [3] AWS Prescriptive Guidance, “Knowledge bases for Amazon Bedrock” (managed RAG, vector stores, Retrieve / RetrieveAndGenerate): https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-fully-managed-bedrock.html

• [4] My series on tool and retrieval design in code, “MCP Strategies on AWS”: https://schristoph.online/blog/mcp-strategies-on-aws-overview/

• [5] Amazon Bedrock User Guide, “Sync your data with your Amazon Bedrock knowledge base” (ingestion jobs, incremental resync behavior): https://docs.aws.amazon.com/bedrock/latest/userguide/kb-data-source-sync-ingest.html

• [6] Amazon S3 User Guide, “Creating a knowledge base with S3 Vectors”: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors-bedrock-kb.html

• [7] Amazon Bedrock User Guide, “Query a knowledge base connected to an Amazon Neptune Analytics graph”: https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-neptune.html

#AI #RAG #VectorSearch #AmazonBedrock #Architecture

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AI
• RAG
• VectorSearch
• AmazonBedrock
• Architecture