How Vector Databases Actually Work
How Vector Databases Actually Work
Every ChatGPT that "reads your docs," every "find me something like this" button — there's a vector database underneath. Here's the whole idea, from numbers to graphs, in plain English.
You know the moment. You type "how do users log in?" into a docs search and it surfaces the OAuth guide, the session-timeout page, and the JWT rotation runbook — even though none of them contain the words you typed. That's not keyword search getting smarter. It's a completely different machine: a vector database. Every RAG chatbot, every semantic search box, every recommendation carousel runs on one.
The good news: the whole thing is built on three ideas, and none of them are scary. Numbers that mean things. Comparing by angle. And a graph that lets you skip the scan. Let me walk you through them.
Step 1: turn everything into numbers
The core trick is called an embedding. You take a piece of data — a sentence, a paragraph, an image, a chunk of audio transcript — and run it through an AI model that outputs a list of numbers. That list captures what the data means, not what it looks like.
Diagrams always draw this in two dimensions because two dimensions fit on a slide. Real embeddings have hundreds or thousands of dimensions: the popular open-source all-mpnet-base-v2 model produces 768 numbers per text chunk, and OpenAI's text-embedding-3-small produces 1,536. Each number encodes some subtle attribute — topic, sentiment, intent — that the model learned during training.
Generating one looks like this, using the standard OpenAI SDK pattern:
import OpenAI from 'openai';
const openai = new OpenAI();
async function embed(text: string): Promise<number[]> {
const response = await openai.embeddings.create({
model: 'text-embedding-3-small',
input: text,
});
return response.data[0].embedding; // 1536 numbers
}
const docVector = await embed('JWT tokens expire after 24 hours');
// [0.023, -0.041, 0.087, ... 1536 numbers]Once your data is numerical, one very useful thing falls out for free: you can find similar items by measuring the distance between vectors. Items that mean similar things land near each other in this high-dimensional space. Items that don't, don't.
Step 2: compare by angle, not by words
"Near each other" needs a ruler. Vector databases typically offer three:
- Cosine similarity — compares the angle between two vectors and ignores their length. This is the usual default for text search: two documents about the same topic point in roughly the same direction, even if one is ten times longer.
- Euclidean (L2) distance — the straight-line distance, Pythagoras generalized to hundreds of dimensions. Used when the magnitude of the vector carries meaning.
- Dot product — the fastest of the three, and works well when your vectors are already normalized to unit length.
The rule of thumb from people who run this in production: check your embedding model's documentation first and use whatever metric the model was trained with. The metric matters more than most tutorials admit — it's part of the deal between the model and the database.
Here's the part that still feels like magic the first time you see it. Take two sentences with almost no words in common — "JWT tokens expire after 24 hours by default" and "how long does authentication last?" — and their embeddings can have a cosine similarity around 0.92, meaning "very similar." A keyword search sees zero overlap. The vector search sees the same meaning from two angles.
Why a normal database can't do this
Your first instinct might be: can't I just index this in Postgres? That's the right instinct — and roughly 80% of the world's data is unstructured, so the question comes up constantly.
The problem is that everything normal databases are good at assumes low dimensions. A B-tree finds the number 42 in a billion rows in logarithmic time — because "42" has a total order. Nearest neighbor in 768 dimensions has no total order. Closeness means simultaneous proximity across all 768 axes, and there is no left-right arrangement of points that preserves that. Tree structures collapse the moment you add a second dimension, let alone 768. This is the famous curse of dimensionality: the usual indexing tricks simply stop working.
The honest baseline, then, is brute force — compare the query against every single stored vector. For a few thousand vectors that's fine. At a million vectors with 768 dimensions each, it's millions of floating-point operations per query, inside a single request. Nobody ships that.
Step 3: the graph that skips the scan
This is where the actual database engineering lives. Instead of scanning everything, modern vector databases build a proximity graph: every vector becomes a node, and each node is wired to a handful of its nearest neighbors. To answer a query, you start somewhere in the graph and greedily hop to whichever neighbor is closer to your target. Because nearby nodes link to nearby nodes, a greedy walk converges on the true neighborhood after a handful of hops — far fewer than the size of the dataset.
The index nearly everyone uses for this is HNSW — Hierarchical Navigable Small World graphs, published by Malkov and Yashunin in 2016. The "hierarchical" part is the clever bit: imagine a skip list. The top layer has a sparse set of long-range links between far-apart points; each layer below gets denser, until the bottom layer contains every vector with short-range links. A query enters at the top, takes long hops to land roughly in the right neighborhood, drops a layer, and repeats — zooming in until the dense bottom layer refines the answer. That's the diagram at the top of this article.
HNSW is now the backbone of FAISS, Milvus, Weaviate, and pgvector — because it offers the best practical trade-off anyone has found between recall and latency.
A vector database is really two things glued together. Embeddings are the representation: lists of numbers that capture meaning. The index is the access path: the data structure that makes finding nearby vectors fast enough to query in production. People come away thinking a vector database is just an array of floats. It isn't — the index is the database part. Everything else — metadata filtering, hybrid search, sharding, backups — is infrastructure built around those two ideas.
One more honest detail: these indexes are approximate nearest neighbor (ANN) algorithms. They trade a small amount of recall for an enormous gain in speed. You're not getting the mathematically perfect top-10; you're getting a very good top-10, thousands of times faster. For semantic search and RAG that trade-off is almost always correct. For something like financial reconciliation, it might not be — and you should know which side you're on.
What else lives in a real vector database
The search is the heart, but production deployments need more than a heart. The features that separate a library from a database:
- Metadata filtering — "find similar transcripts, but only from this account, from the last 90 days." Similarity plus structured predicates, evaluated together.
- Namespaces / collections — partitions inside one index so multi-tenant data never leaks across customers, or so dev and prod stay separate.
- Real-time upserts — add, update, and delete vectors without rebuilding the whole index. HNSW supports incremental inserts, which is one reason it beat cluster-based methods like IVF for live workloads.
- Hybrid search — combining keyword (BM25) and vector scores, so exact product codes still match exactly while everything else matches semantically.
The vendor landscape in 2026 splits three ways: managed vector-native services like Pinecone (zero index operations), open-source self-hosted engines like Qdrant, Weaviate, Milvus, and Chroma, and the "add vectors to the database you already run" camp — pgvector for Postgres, plus vector search in Redis, Elasticsearch, and OpenSearch. The honest advice that keeps showing up from people running this: pick on operational fit, not benchmarks. Every serious option does approximate nearest neighbor well enough that your retrieval quality is determined by your embeddings and your chunking — not by the index.
The honest question: do you actually need one?
Probably not on day one. That's the part of this topic that doesn't get said enough. For many projects a dedicated vector database is overkill: a traditional database with a vector extension works fine, so does an in-process library — and if you have a few thousand vectors, computing cosine similarity in a NumPy array in memory is a legitimate architecture, not a hack.
Reach for the dedicated database when one of these is true: your corpus is large enough that exact search blows your latency budget (roughly hundreds of thousands of vectors and up); you need metadata filtering alongside similarity; you need real-time upserts instead of a nightly rebuild; or you need durability, replication, and someone else's on-call rotation. If none of those apply yet, start with pgvector in the Postgres you already run. The embeddings are portable, and the index is the part you're throwing away anyway.
Frequently asked questions
What is a vector database in simple terms?
A database that stores data as lists of numbers (embeddings) capturing meaning, and retrieves items by semantic similarity instead of exact keyword match. Ask "how do users log in?" and it finds the OAuth docs — no shared words required.
How is a vector database different from a regular database?
Regular databases store structured rows and match on exact values. Vector databases store high-dimensional embeddings and match on closeness in meaning, using distance metrics like cosine similarity. The two can coexist — pgvector literally adds vector search to PostgreSQL.
What is HNSW and why does every vector database use it?
HNSW (Hierarchical Navigable Small World) is a graph-based index published in 2016 that answers "find the nearest vectors" by hopping through layers of progressively denser graphs. It turns a scan of millions of vectors into a walk of dozens of hops — the best practical recall/latency trade-off the field has found.
What are vector databases used for?
Four patterns cover most real deployments: RAG (giving chatbots long-term memory over your own documents), semantic search, similarity search over images/audio/video ("find me something like this"), and ranking/recommendation systems.
Do embeddings store my original data?
No — and this matters. An embedding is a one-way transformation: you can compare two embeddings for similarity, but you can't reconstruct the original text from one. That's why every vector database pairs each vector with metadata (the original text, source URL, timestamps) that gets returned with the results.
The bottom line
Strip away the hype and a vector database is a beautifully simple deal: convert meaning into numbers, compare the numbers by angle, and use a clever graph so you never have to compare them all. Embeddings give you the representation, HNSW gives you the speed, and everything else is operational plumbing.
The reason this topic matters now is that the index has quietly become a commodity. Vector search ships in Postgres, Redis, Elasticsearch, Cloudflare's edge — it's infrastructure you switch on, not a decision you agonize over. What hasn't commoditized is the part nobody talks about: the quality of the data you put in. Tune your chunking, mind your embeddings, and keep your metadata honest. The database will do the rest in under a millisecond.
Sources and further reading
- AssemblyAI — "What is a vector database? Embeddings, indexes, and RAG over audio transcripts"
- dev.to — "Vector Databases Explained: Embeddings, Similarity Search, and When to Use Them"
- HNSW-Sharp docs — "What is HNSW?" (intuition, scaling, trade-offs)
- Malkov & Yashunin (2016) — "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs"
- Qdrant — "What is a vector database?" (source of the HNSW diagram above)
