What is a vector database?

How a vector database works

Ordinary search matches words. A vector database matches meaning, by working with embeddings: lists of numbers an embedding model produces so that texts with similar meaning end up close together. The flow has two halves.

• Indexing. Split your documents into passages, run each through an embedding model, and store the resulting vectors with metadata — source, date, permissions — and the passage itself or a pointer to it. The database organises the vectors into an index built for fast lookup.
• Querying. Embed the user's question with the same model, ask the database for the nearest vectors, optionally filter by metadata (only this customer's files, only this year), and get back the top few passages, ranked by similarity.

Because closeness is measured in meaning, a question about “cancelling my plan” can find a passage titled “ending your subscription” that shares no keywords with it.

Vector database, vector index or ordinary database

Three things get called by this name. A vector index library, such as FAISS, does the fast similarity search and nothing else: you manage storage, updates and backups yourself, and changing the data can mean rebuilding the index. A vector database wraps that search in the duties of a database — inserting, updating and deleting records as they change, filtering on metadata, scaling, backups, access control and separating customers' data. And an ordinary database with vector search adds the capability to a system you already run: the open-source pgvector extension, for example, gives PostgreSQL vector columns, exact and approximate nearest-neighbour search and the usual transactions and joins alongside them.

The choice is mostly operational. If your data already lives in Postgres and fits on one machine, adding vectors there avoids a second system to keep in sync. A dedicated service earns its place at large scale, under heavy write traffic, or when you would rather not run the infrastructure.

Approximate search and its trade-off

Comparing a query with every stored vector gives an exact answer, but the work grows with the size of the collection. Vector databases instead use approximate nearest-neighbour indexes that check only a small, promising part of it:

• HNSW builds a layered graph of neighbours and walks it toward the query — fast and accurate, at the price of memory and build time.
• IVF groups vectors into clusters and searches only the clusters nearest the query — cheaper to build, somewhat slower to query.
• Product quantization compresses vectors into short codes so far more of them fit in memory, trading a little precision.

All of them give up a small amount of recall — occasionally missing a true nearest neighbour — for a large gain in speed. Closeness itself is measured by cosine similarity, dot product or Euclidean distance; the right one is usually whichever the embedding model was trained for.

Where it fits in RAG — and why it saves tokens

In retrieval-augmented generation, the vector database decides what the language model gets to read. Instead of pasting a whole manual into every prompt, the application retrieves the handful of passages most related to the question and sends only those. That improves answers by keeping the model focused, and it changes the bill at least as much: the model's input shrinks from the size of the corpus to the size of a few passages.

The worked example puts numbers on it for a support assistant answering from a 150,000-token product manual, once with the whole manual in every prompt and once with five retrieved passages. The alternative to retrieval is not always that crude — a large context window with the manual held in the prompt cache narrows the gap — but cached tokens are still billed on every request.

What a vector database costs

Three separate meters add up to the cost of semantic search:

• Embedding tokens. Every passage is embedded once when indexed and again whenever it changes; every query is embedded once. Embedding models are among the cheapest per token in our index — the panel below shows current rates — so this is usually the smallest line, except when you re-embed a whole corpus after switching embedding models, which you must, because vectors from different models are not comparable.
• The database itself. Managed services meter what you store and how hard you use it. Pinecone's serverless plans, for example, bill storage per gigabyte-month, writes and reads in metered units, and data transferred out, with a free starter tier capped in each of those and paid tiers that carry a monthly minimum. Self-hosting swaps those meters for the servers, memory and maintenance.
• The tokens you retrieve. Retrieved passages become input to the language model on every question, so the number and length of passages you send is a pricing decision as much as a quality one.

Do you need one?

Not always. A few hundred documents can be embedded and searched in memory with a library, or skipped entirely by putting them in a cached prompt. A team already on Postgres can add pgvector before adopting anything new. A dedicated vector database pays off when the collection is large, changes constantly, needs filtering by user or permission, or has to answer many queries a second — the point at which running search yourself stops being a side task.