RAG Pipeline for Student Documents
Retrieval-augmented generation turns messy course files into ranked evidence before an AI tutor answers.
Evidence, retrieval, agents, and the paths that connect a question to a grounded answer.
Interactive model
Ranked evidence before generation
Move from a student question to candidate chunks, hybrid scores, and the evidence that should enter the answer context.
Live HTML simulation · adjust the controls and watch the computed output respond.
Interactive
Hybrid retrieval turns a vague study question into ranked evidence
This is a simplified teaching model. Its displayed values are computed from the controls; the article explains where the model stops.
Site connection
Lykke describes document ingestion, embeddings, hybrid search, reranking, and Canvas-connected study workflows.
What RAG Adds
A language model can produce fluent text from its parameters, but those parameters do not automatically contain a student's current syllabus, revised rubric, or professor-specific notation. A retrieval-augmented generation (RAG) system first finds passages from an indexed collection and then supplies selected passages as context for generation.
For course tools, evidence is fragmented across Canvas pages, assignments, files, grades, syllabi, calendars, textbooks, notes, and PDFs. Retrieval makes those pieces addressable without pretending that every stored passage is equally relevant.
Mental model: retrieval is an open-book evidence selection step; generation is the explanation step. The analogy stops when retrieval misses, misparses, or selects stale evidence—the model cannot read a page that never entered its context.
The Working Pipeline
The document is parsed into chunks, each chunk is embedded, and metadata such as course, week, file name, and assignment date is attached.
At question time, keyword search catches exact terms while vector search catches semantic matches. A reranker can then reorder the mixed candidate set before the model sees it.
| Stage | Job | Failure if skipped |
|---|---|---|
| Chunking | Split source material into retrievable units | The model gets either too little context or a huge noisy passage |
| Embedding | Represent meaning as vectors | Semantic questions miss relevant notes |
| Keyword search | Preserve exact names, formulas, and course terms | Acronyms and proper nouns disappear |
| Reranking | Put the best evidence first | The answer uses convenient but weak context |
Worked Example
Suppose a student asks, 'What should I study before the vector search quiz?' The phrase 'vector search quiz' gives keyword retrieval an exact anchor, while its meaning points vector retrieval toward passages about embeddings, similarity, hybrid search, and reranking.
The first stage might return a Canvas quiz entry, two lecture chunks, an older announcement, and a study-guide checklist. Course and date metadata remove out-of-scope material, deduplication collapses repeats, and the reranker promotes the current quiz entry, lecture, and guide.
The answer can then separate claims by source: timing from Canvas, terms from the lecture, and practice prompts from the guide. If no source states which formulas are assessed, the response should say so rather than inventing coverage.
Candidate Fusion and Context Assembly
Keyword search protects exact names, formulas, course codes, and assignment labels; vector search can recover passages phrased differently. Hybrid retrieval collects candidates from both signals, but their raw scores are not automatically comparable, so the system needs a defined fusion rule.
Deduplication preserves room for distinct evidence. A reranker evaluates a smaller candidate set against the full question, then context assembly retains source labels and fits the strongest passages into a bounded prompt. More chunks are not automatically better: unrelated context can distract generation.
A reranker can reorder only the candidates it receives. If first-stage retrieval misses the evidence, reranking cannot recover it.
Evaluation Before Deployment
Evaluate retrieval separately from answer quality. For a small set of real student questions, label the minimum sufficient source chunks and measure whether they appear in the candidate set and near the top after reranking.
Then evaluate grounded generation: every material claim should follow from supplied evidence, source labels should survive synthesis, and the answer should qualify or abstain when the sources are incomplete. Test stale assignments, duplicate uploads, ambiguous course codes, scanned PDFs, and questions spanning several files.
| Layer | Question | Failure |
|---|---|---|
| Parsing | Was source content recovered faithfully? | A formula or due date disappears |
| Retrieval | Did sufficient evidence enter the candidates? | The current rubric is absent |
| Ranking | Did the best evidence rise high enough? | An old announcement wins |
| Generation | Does each claim follow from context? | The answer invents an exam topic |
Limits and Common Misconceptions
RAG does not make a model truthful by definition. Parsing errors, incoherent chunks, retrieval misses, stale indexes, malicious instructions inside uploaded documents, and unsupported synthesis can still produce a wrong response.
Vector similarity is not proof of relevance, reranking is not fact checking, and a citation does not prove that the cited passage supports the adjacent sentence. Course tools also need authorization boundaries so material from one user or course cannot leak into another retrieval scope.
The project source describes both production cloud retrieval and browser-side RAG work. These imply different privacy, latency, storage, and hardware tradeoffs; the source does not establish that either architecture is always preferable.
Common Pitfalls
- Treating embedding search as enough when exact course terms matter.
- Chunking by arbitrary character count instead of document structure.
- Letting retrieved chunks into the prompt without source labels.
- Using old course files after Canvas content changes.