RAG or fine-tuning? Choosing the right approach for an enterprise knowledge assistant

Every company that has tried a general chatbot on its own documents has met the same problem: confident answers that are wrong. Two techniques fix it. Retrieval-augmented generation (RAG) gives the model your documents at question time. Fine-tuning changes the model itself. They are often confused, rarely interchangeable, and the wrong choice costs months.

01What RAG does

RAG stores your documents as searchable chunks, finds the most relevant ones for each question and passes them to the model together with the question. The model writes an answer grounded in those passages and can cite them. Update a document and the next answer reflects it. Nothing about the model changes.

RAG is the right default for knowledge assistants: policies, manuals, contracts, product specifications, support history. It is transparent, cheap to keep current and auditable, which matters under GDPR and for any regulated activity.

02What fine-tuning does

Fine-tuning trains a model further on your examples so it adopts a style, a format or a specialised behaviour. It does not reliably teach facts, and it cannot cite sources. Use it when you need consistent output in a strict format, a domain-specific tone, or a smaller, cheaper model that behaves like a larger one on a narrow task.

Fine-tuning is a build-and-maintain commitment. Each change to the behaviour means new training data, a new run and a new evaluation. For most knowledge use cases it is unnecessary.

03The decision in one table

• Answers must reflect documents that change weekly: RAG.
• Answers must cite a source for compliance: RAG.
• Output must follow a rigid template thousands of times a day: fine-tuning, often on top of RAG.
• You need a small model on your own servers for cost or privacy: fine-tuning a small open-weight model, fed by RAG.
• You have fewer than a few thousand high-quality examples: RAG first, fine-tune later if ever.

04What makes RAG work in practice

The model is rarely the problem. Retrieval quality is. Chunking that respects document structure, metadata filters (department, date, product), hybrid search that combines keywords and embeddings, and a re-ranking step make the difference between a demo and a system people trust. Qdrant, pgvector and similar stores are mature; the engineering effort is in the pipeline around them.

The second ingredient is evaluation. Build a set of 100 to 300 real questions with approved answers, run it on every change, and publish the score to the people who use the assistant.

05Data protection questions to settle first

• Which documents may be indexed, and who may see which answers? Access control belongs in the retrieval layer, not in the prompt.
• Where do embeddings and prompts travel? Regional endpoints or self-hosted models keep data inside the EU, Singapore or Vietnam when required.
• How long are questions and answers logged, and for what purpose? Write it down before launch.

06A typical first project

Six weeks: two to connect sources and build the pipeline, two to tune retrieval against the evaluation set, two to integrate with the channel your team uses (Teams, Slack, the intranet or the help desk) and to add access control. The result is an assistant that answers with sources, and a scorecard that tells you how good it is.

07How Synova helps

We design and build RAG pipelines on Claude, OpenAI, Gemini or open-weight models, choose the vector store and hosting that fit your data constraints, and run the evaluation loop with you. Fine-tuning is added only when the numbers show it is needed.