Design a RAG Assistant

Intermediate · 26 min read · High-Level Design, ML & AI Systems

Answering from company documents: chunking, hybrid retrieval, permissions, grounding, injection and evaluation.

Design an AI assistant that answers employees' questions using the company's internal documents (wikis, design docs, policies, tickets), with links to its sources: "What's our on-call escalation policy for payments?" or "How do I get access to the analytics warehouse?".

The technique is retrieval-augmented generation (RAG): find the passages relevant to the question, put them in the prompt, and have a large language model (LLM) answer from them.

1. Clarify the problem

2. Why RAG, not fine-tuning

Fine-tuning still has a place, for example to teach the model the company's tone or answer format, but facts belong in retrieval.

3. Capacity estimation

4. High-level design

5. Deep dives

Retrieval can't find what ingestion mangled, so this stage deserves real attention.

6. Failure modes

7. Trade-offs summary

8. What interviewers look for

Interview questions

  • Why not just put all the company's documents into a model with a very long context window?
  • How do you make sure the assistant never reveals a document the user can't access?
  • How do you evaluate a RAG system when answers are free text?
  • What is indirect prompt injection and how do you defend against it?

Practice

Write your answer before opening the suggested one. Anything you mark "Partly" or "Missed it" comes back in your review deck.

Further reading

Key takeaways

  • RAG answers from retrieved passages: fresh, citable and permission-aware; fine-tuning suits style and format, not changing facts.
  • Ingestion quality decides retrieval quality: structure-aware chunks, title and section prefixes, metadata, fast updates and deletes.
  • Retrieve with hybrid dense + keyword search, then re-rank a shortlist with a cross-encoder.
  • Enforce permissions as a filter inside retrieval; never trust the prompt to protect documents.
  • Ground answers with citations, abstain when sources are weak, and stream the response.
  • Defend against indirect prompt injection by treating passages as data and limiting what the assistant can do.
  • Evaluate retrieval and generation separately with a golden set and calibrated LLM judges.

Previous: Design Ad Click Prediction · Next: Design an LLM Serving Platform