Should You Actually Build a RAG System? A Decision Guide

Every team building anything with an LLM eventually asks: "Should we add RAG?" The honest answer in 2026 is "probably yes, but not the way most people start." Retrieval-augmented generation looks deceptively simple in a tutorial and surprisingly hard in production. Here is when to build one and when to do something simpler.

What RAG actually is

At its simplest: when a user asks a question, retrieve relevant chunks of your knowledge base, paste them into the LLM's context, and ask the LLM to answer based on the retrieved content. Three components:

1. An embedder. Converts text into vectors. OpenAI's text-embedding-3-small, Cohere Embed, or open-source models like BGE.
2. A vector store. Stores embeddings and supports nearest-neighbour search. Pinecone, Weaviate, Qdrant, pgvector on Postgres.
3. An LLM. Takes the retrieved chunks plus the user's question and generates an answer.

The first version of this can be built in a weekend. Productionising it takes weeks.

When RAG is the right answer

• Your knowledge changes faster than you can fine-tune. Documentation, internal wikis, product catalogs that update weekly.
• You need to cite sources. Compliance, legal, medical contexts where the answer must trace back to a specific document.
• Your knowledge is too large to fit in a prompt. Anything more than ~50–100 pages.
• You want different users to see different information based on permissions or context.

When something simpler is enough

• Stuffing the prompt. If your entire knowledge base fits in 50K tokens (about 100 pages), just include it in the system prompt. Modern context windows are large enough that this works fine for many use cases.
• Fine-tuning. If your knowledge is stable and you want consistent behaviour, fine-tuning a smaller model on your data can be cheaper at runtime than running RAG.
• Plain old search. If the user's actual question is "find me the document that mentions X," a good full-text search engine (Postgres FTS, Meilisearch, OpenSearch) is faster and more accurate than RAG.

The production realities most demos hide

Chunking is harder than it looks

Splitting documents into chunks of 500–1000 tokens is the default starting point. It is also why most RAG systems perform poorly. Half-sentences split across chunks lose meaning. Tables and code blocks break. Hierarchical structure (sections, subsections) is lost. Spend real time on chunking strategy — semantic chunking, parent-document retrieval, or document-aware splitting often improve quality more than swapping the embedder.

Retrieval quality is the bottleneck, not generation

If the right chunks are in the LLM's context, almost any modern LLM will answer well. If they aren't, no amount of prompt engineering recovers. Profile every failure: was the right chunk retrieved? If yes, was it ranked high enough? If yes, did the LLM still misuse it? Most failures are at step 1 or 2.

Reranking helps a lot

Two-stage retrieval — embed-based recall (fetch top 50), then a cross-encoder rerank (rerank to top 5–10) — typically improves quality more than any other single change. Cohere Rerank, BGE Reranker, or commercial alternatives all work.

Hybrid search is usually better than vector search alone

Vectors are great for semantic similarity ("how do I authenticate?" matches "OAuth setup"). Keywords are great for exact identifiers ("error code E-1042"). Real systems combine both with reciprocal rank fusion or weighted scores.

Evaluation is non-negotiable

Without an eval suite (50–200 sample queries with expected answers), you can't tell whether changes are improvements or regressions. Build the eval suite before building the third RAG iteration. Frameworks: ragas, deepeval, or hand-rolled.

Citations and grounding

Users trust RAG systems more when they can see the source. Always return the document IDs of retrieved chunks alongside the answer; UI should let users click through.

The honest cost

A production-grade RAG system for a typical SaaS knowledge base — say, 10,000 documents, 1 million chunks, 1,000 queries per day — costs roughly:

• One-time: 4–8 engineering weeks for the first version, plus 2–4 weeks for evaluation infrastructure and iteration.
• Monthly: ₹15,000–₹40,000 for vector database hosting, ₹20,000–₹60,000 in LLM API costs, embedding refresh as data changes.
• Ongoing: Quality regression monitoring, prompt tuning when models update, content pipeline maintenance.

That isn't enormous, but it isn't free. Don't build RAG to demo-impress investors. Build it when there is a concrete business problem RAG solves better than alternatives.

The takeaway

RAG is the right tool when knowledge is large, changes often, and citations matter. It is the wrong tool when knowledge is small enough to fit in a prompt, stable enough to fine-tune, or simple enough that search would do. Build the simpler thing first. Move to RAG when the simpler thing demonstrably can't keep up.