🤖 AI / ML

Retrieval-Augmented Generation (RAG)

RAG · Grounded Generation · Retrieval-Augmented LLM

Definition

RAG (retrieval-augmented generation) is an AI architecture in which a language model is extended with a retrieval step: relevant documents are pulled from an external store and passed into the LLM context so the answer is built from real data.

How RAG works

RAG solves a fundamental problem with LLMs: a language model knows the world as of its training
cut-off and has no access to your specific catalogue, your prices or your stock. RAG adds a
retrieval step over a current knowledge base.

The request pipeline:

User request → Request embedding
                    ↓
             Vector search over the index (Top-K documents)
                    ↓
   LLM: [system prompt] + [retrieved documents] + [request]
                    ↓
              Answer to the user

All of it happens inside a single request latency — typically 500–2000 ms for the full cycle.

Components of a RAG system

Indexing (offline):
– Catalogue documents are split into chunks — structured fragments such as a product record, an
FAQ entry or a category description.
– Each chunk is converted into a vector by an embedding model.
– The vectors are stored in a vector database alongside their metadata (price, availability,
category).

Retrieval and generation (online, on every request):
– The user’s request is converted into a vector by the same model.
– The system finds the Top-K nearest vectors — usually 5–10 documents.
– Those documents plus the request are passed to the LLM as context.
– The LLM generates an answer grounded in the supplied data.

Important: the quality of RAG is set by the quality of the retrieval step. If the right
document never makes it into the Top-K, no LLM is powerful enough to give the right answer.

RAG in e-commerce

In retail, RAG addresses three jobs:

Job	Without RAG	With RAG
“Do you have the iPhone 15 Pro 256 GB in stock?”	Hallucination, or “I don’t know”	Real availability checked against the catalogue
“Recommend a mattress for back pain”	Generic advice with no link to the assortment	Specific models with prices from the catalogue
“How does model A differ from model B?”	Specifications get mixed up	A comparison built from the real product records

Limits and practical caveats

Context size: only a limited number of documents fits into the prompt. On a large catalogue,
ranking quality at the retrieval step becomes critical.
Freshness: the knowledge base has to track catalogue changes — prices, availability — which
in e-commerce usually means in real time.
Handling an empty result: when retrieval finds nothing relevant, the model should say so
honestly rather than invent something similar.

Frequently asked questions

Why use RAG when you could fine-tune an LLM on the catalogue?

Fine-tuning bakes knowledge into the model weights — expensive and slow. The moment the catalogue changes (new products, new prices) the model is out of date. RAG reads current data on every request: retrieval picks the relevant documents in real time and the LLM answers from them. Updating the knowledge base takes minutes, not days of retraining.

How does RAG prevent hallucination?

An LLM with no context fills in the blanks — prices, specifications, availability — from statistical patterns in its training. RAG puts concrete fragments from the real catalogue into the prompt, so the model answers from that data rather than from its weights. With the right system-prompt instruction the model says it does not know when the product is not in the knowledge base.

What is vector search in the context of RAG?

Catalogue documents are converted into vector representations (embeddings) and stored in a vector database. When a shopper asks something, their text is turned into a vector too, and the system finds the documents closest in meaning — even when the words do not literally match. That is how “shoes for a marathon” retrieves running trainers.

How large a knowledge base can RAG handle?

Modern vector databases (Pinecone, Weaviate, pgvector) scale to millions of documents, so a catalogue of 100,000 SKUs is nowhere near the ceiling. The parameters that actually matter are embedding quality, chunk size, and how good the metadata (price, category, availability) is for filtering before generation.

Gravity Field