Vector Store: How Your AI Gets a Memory

Imagine you want an AI to know your 80-page club bylaws. Or your recipe collection. Or the calorie tables you feed your nutrition assistant.

The naive solution: paste everything into the chat. Works — until the document gets too big, the model forgets the beginning, and every query becomes needlessly expensive.

The elegant solution is called a vector store. And it's easier to understand than the name suggests.

The core problem: AI has no memory

A language model knows nothing about you. Every new chat starts from scratch. You can paste in context, sure — but space is limited (the so-called context window), and sending an entire manual with every question is like hauling the whole library along just to look up a single page.

So there has to be a way to hand the AI exactly the right bit and nothing else. That's precisely what a vector store does.

The idea: meaning becomes numbers

A computer doesn't understand words, only numbers. So a vector store translates every piece of text into a long string of numbers — a so-called embedding. The trick: those numbers capture the meaning, not the letters.

Texts with similar meaning get similar numbers and end up close together. "Dog" and "four-legged friend" sit close. "Dog" and "tax return" land far apart.

Picture a library where the books aren't shelved alphabetically but grouped by topic — everything about cooking together, no matter what the titles say. You're not searching for an exact keyword, you're searching for sense.

A worked example you can check by hand

This is where it gets a little more technical — but don't worry: it all comes down to multiply-and-add, the kind of thing any calculator handles. Feel free to skip it; the principle stands without the numbers.

Let's take five meaning dimensions — five themes we measure a text against: cooking, nutrition, fitness, gear, money. Each text gets a value per theme, higher the more it fits. (Real embedding models find these values themselves — here we set them by hand.)

Two documents:

Those five numbers are the document's vector. To compare only the direction (not whether a text simply has more words), we scale each vector to length 1 — that's called normalizing. You divide by the vector's own length:

• length of A = √(3² + 4² + 1²) = √26 ≈ 5.10 → A ≈ (0.59, 0.78, 0, 0, 0.20)
• length of B = √(1² + 4² + 2² + 1²) = √22 ≈ 4.69 → B ≈ (0, 0.21, 0.85, 0.43, 0.21)

Now the search. Your query becomes a (normalized) vector too, depending on which themes it stresses. And because every vector has length 1, the cosine similarity is just the dot product: multiply the values pairwise, add them all up. A result near 1 means "points the same way" — very similar; near 0 means "little in common."

Why cosine in the first place? Because it's simple to compute and stays intuitive: a large cosine means a small angle between the vectors — they point almost the same way. There are other similarity measures too, but cosine is the classic.

Worked out once for the query "low-calorie cooking" = (1, 1, 0, 0, 0), normalized (0.71, 0.71, 0, 0, 0):

• with A: 0.59 · 0.71 + 0.78 · 0.71 = 0.97
• with B: 0.21 · 0.71 = 0.15

Doc A wins clearly. Here it is across four queries:

The last row is the interesting one: "nutrition" alone would point to Doc A — but "training" pulls harder toward Doc B, and B wins overall. The store doesn't match keywords; it weighs the whole meaning.

And that's the entire trick: just multiply and add. What we do here for 2 documents and 5 dimensions with a calculator, a server does for millions of documents and thousands of dimensions — in milliseconds. Real embedding models typically use 500 to 4,000 dimensions instead of five. The principle is exactly the same.

How it comes together: RAG

The technical term is RAG — retrieval-augmented generation. Sounds clunky, but it just means: before answering, the AI fetches the relevant morsels from your knowledge base.

In four steps:

1. Split: Your document is cut into small morsels ("chunks") — roughly paragraph by paragraph.
2. Embed: Each morsel gets its numerical fingerprint and goes into the vector store.
3. Search: When you ask a question, it's turned into numbers too — and the store returns the most similar morsels.
4. Answer: Only those few morsels, plus your question, go to the model.

In practice, by the way, you rarely pull just the single best hit — depending on the use case, you grab the top three to five. Especially with thousands or millions of documents in the store, that keeps anything important from slipping through. It doesn't change the principle.

Instead of sending the whole book, only the relevant page goes out. That saves money and makes the answer more concrete — it leans on your sources rather than on whatever the model happened to read somewhere once.

Nice side effect: less hallucinating. When the AI has the right passage right in front of it, it makes things up less often.

What you actually need

The good news: you don't assemble the vector store yourself. Ready-made tools handle the splitting, embedding, and searching for you:

• OpenWebUI has a knowledge base built in — drop a PDF in, done.
• Flowise lets you click together entire AI workflows, no code.
• Qdrant is the vector store itself — runs cleanly in Docker if you'd rather keep it in your own hands.

And here's the genuinely nice part: all of this runs on your machine. Your documents never leave your own computer — no cloud, no third party reading along. For anything confidential, that's a real argument.

A quick history lesson: none of this is new

Sounds like brand-new AI magic? It isn't. Sorting texts as vectors in a space is older than most readers of this blog — the idea goes back to the Vector Space Model of the 1960s and 70s. Big search and library systems already worked this way, and around the turn of the millennium web search engines ranked their hits on exactly this principle.

So the vector store isn't the new part. What's new is how the numbers come about. People used to simply count words and weight them with formulas like TF-IDF (known in the SEO world as WDF*IDF): the rarer a word is overall, the more important it is for this one document. Pure bookkeeping over words — no trace of meaning.

Today a language model does the embedding. It has learned that "dog" and "four-legged friend" belong together without the same word ever appearing. That was impossible before — and that's exactly why the old idea is enjoying a second spring.

When it's worth it — and when it isn't

Honestly: for a quick question to ChatGPT, you don't need a vector store. It pays off when you keep working against the same body of knowledge — your own notes, manuals, a documentation set, your recipe collection. Anything too big to re-explain every time.

For one-off use it's overkill. For a memory that grows with you, it's exactly right.

How to get started

The easiest entry point is OpenWebUI: create a knowledge base, drop in your documents, assign it to a custom model — the tool does the rest. If you want to go deeper, start Qdrant with a single Docker command and feel out what searching by meaning is like.

This is the kickoff for a small series on building your own AI systems. Next up, we'll look at Qdrant in concrete terms — and how it turns into a chatbot that actually knows your content.