Retrieval-Augmented Generation grounds LLM output in your specific documents, drastically reducing hallucination and letting the model answer questions.
Retrieval-Augmented Generation (RAG) is the pattern where, at query time, relevant chunks of a source corpus are retrieved and inserted into the LLM prompt as context. The model then answers using that context rather than its training knowledge alone. It is how you get an LLM to accurately answer questions about your product docs, your customer's data, or last week's board memo. Getting RAG to production quality is 90% retrieval quality and 10% clever prompting — a fact many teams learn the hard way.
Ingest: source documents (PDFs, HTML, transcripts, database records) are cleaned, parsed, and split into chunks. Embed: each chunk is converted to a dense vector via an embedding model (OpenAI text-embedding-3, Cohere embed, Voyage, open-source BGE / E5). Store: vectors are written to a vector database (pgvector on Postgres, Pinecone, Weaviate, Qdrant, Turbopuffer) alongside metadata. Retrieve: at query time, the user question is embedded and the top-K nearest chunks are returned. Re-rank: a cross-encoder re-orders the top-K for better precision. Generate: retrieved chunks + question go into the LLM prompt; the model answers grounded in the context. Every stage has failure modes; instrument and measure each.
The chunk is the unit of retrieval; get it wrong and no downstream cleverness rescues quality. Naive fixed-size chunking (500 tokens with 50 overlap) is a starting point, not a solution. Better: (1) Structural chunking — split on document structure (headings, sections, code blocks) so a chunk is semantically self-contained. (2) Sentence-aware — never split mid-sentence. (3) Include heading context in each chunk ('Section 3.2: Refund Policy — <chunk text>') so the retrieved snippet makes sense on its own. (4) Preserve tables as single chunks. (5) Store parent-child relationships — retrieve small chunks, feed larger parent windows to the LLM. Chunking is document-format-specific; PDFs, code, and chat transcripts all need different strategies.
Pure vector search is bad at proper nouns, part numbers, exact phrases, and rare tokens — the exact things users search for when they know what they want. Hybrid retrieval: run both vector search (semantic) and BM25/keyword search (lexical); combine results via reciprocal rank fusion or a weighted score. Postgres with pgvector + tsvector, or Elasticsearch/OpenSearch with a vector plugin, natively support this. Almost every serious RAG system in production is hybrid; vector-only is a demo pattern that plateaus.
Retrieve top 20-50 candidates with hybrid search (fast, coarse). Re-rank them to top 5-10 with a cross-encoder (Cohere Rerank, Voyage Rerank, BGE-reranker) — slower per pair but massively more accurate because the model sees query and passage together. The uplift is often larger than any prompt-engineering trick. Cost: fractions of a cent per query at moderate volume. Skip re-ranking only if latency budget is under 500ms end-to-end.
Measure retrieval and generation separately. Retrieval: for a golden set of query-to-relevant-chunk pairs, compute recall@K and MRR — did the correct chunk end up in the top K? If not, no LLM prompt fixes it. Generation: given a good retrieval, does the LLM answer correctly and faithfully? Use LLM-as-judge with a rubric that penalizes unsupported claims. Common wins uncovered by systematic evals: change chunking strategy (20-40% recall lift), add hybrid search (10-20% lift), add re-ranking (10-20% precision lift), tune the prompt for citation ('quote the source passage before answering') to reduce hallucination.
Investor directory · Fundraising library · Articles A–Z · Company funding database