Evaluated RAG assistant
Build an assistant that answers from your documents with citations, abstains when unsure, and is measured on an evaluation set.
Brief ready; automated checks coming soonAdvancedAbout 8 hoursRAG engineeringLLM engineering
What you'll build
- Answers cite the chunks they use
- The assistant says when the documents don't contain the answer
- An evaluation set of at least 20 questions measures retrieval and answer quality separately
- A failure report classifies wrong answers by pipeline stage
Milestones
Work through these in order. Each one ends with a check you can verify yourself.
- 1Milestone 1: RetrievalReuse or build a retriever with measured recall.Done when: Recall@k reported on the evaluation set.
- 2Milestone 2: Grounded generationPrompt the model to answer only from context and cite sources.Done when: Every claim in sampled answers maps to a cited chunk.
- 3Milestone 3: AbstentionHandle questions the documents can't answer.Done when: Unanswerable questions get an explicit "not in the documents" response.
- 4Milestone 4: Failure analysisGroup failures by cause and propose fixes.Done when: A report with counts per failure type.
Test criteria
What a finished project must demonstrate.
- Retrieval and answer quality are scored separately
- Unanswerable questions are included in the evaluation set
- Citations point to chunks that actually contain the cited claim
An automated runner for these criteria is planned. Until it ships, check each one yourself; nothing here is graded for you yet.
Rubric
Rate your work against each criterion. Exemplary isn't required; be honest about where you are.
Reflection
A few minutes of reflection makes the lessons stick.