Vector Search vs Keyword Search in Study Tools
Keyword search finds exact words; vector search finds nearby meaning. Strong study systems usually need both.
Visual explainer
Two search engines. One answer.
Tap a signal to see what it contributes. Hybrid search keeps the precision of exact terms and the intuition of meaning.
A lexical index locks onto identifiers, names, acronyms, and precise phrases. It is the fast, literal signal.
Interactive model
Two signals, one ranked evidence set
The ranked evidence demo shows why exact keyword matches and vector similarity should both influence retrieval.
Live HTML simulation · adjust the controls and watch the computed output respond.
Interactive
Hybrid retrieval turns a vague study question into ranked evidence
This is a simplified teaching model. Its displayed values are computed from the controls; the article explains where the model stops.
Site connection
Lykke uses document embeddings, hybrid search, and reranking to retrieve course evidence before generating study artifacts.
Two Kinds of Relevance
Keyword search is lexical. It cares whether the same tokens appear. Vector search is semantic. It cares whether two chunks mean similar things under an embedding model.
Why Exact Matching Still Matters
Keyword search is lexical: it indexes terms and finds records containing matching tokens or phrases. When a student asks for 'CS 111 Project 2 rubric,' the course code, project number, and rubric label should remain strong signals.
Full-text indexes such as SQLite FTS5 support token-based queries and ranking. Their strength is literal precision, not human-like understanding, so vocabulary mismatch remains a predictable failure mode.
Why Semantic Search Matters
Students rarely ask in the words used by a slide deck. A lecture may say 'gradient-based optimization' while the student asks 'how does the model learn from error?' Embeddings can connect those meanings, although proximity is not proof that a passage answers the question.
The Lykke source describes vector similarity plus keyword hybrid search, followed by reranking and source-linked study artifacts.
| Query type | Useful first signal |
|---|---|
| Exact assignment name | Keyword |
| Vague conceptual question | Vector |
| Formula with notation | Keyword plus metadata |
| Study-plan request | Vector plus calendar metadata |
| Acronym or abbreviation | Keyword |
Worked Example
For 'Why does the optimizer move opposite the slope?', a chunk titled 'Gradient Descent Update Rule' may never use that exact phrase. Vector search can surface the paraphrased relationship even when lexical overlap is thin.
For 'Find the CS 210 gradient descent lab rubric,' semantic retrieval may return many explanations, while lexical matching favors the exact course code, lab name, and word 'rubric.' Hybrid retrieval collects both; current-course metadata and reranking should put the rubric first.
Lexical relevance and vector similarity may use unlike scales, so their scores should not be averaged blindly. Fusion must be explicit, overlap deduplicated, and final evidence inspected.
Keep the query and corpus fixed while comparing lexical, vector, and hybrid modes; otherwise differences may come from the test setup.
How Hybrid Ranking Is Built
A hybrid system runs lexical and vector retrieval and receives two ranked lists. A documented normalization or rank-fusion rule combines candidates without assuming raw scores mean the same thing.
Deduplication removes repeats, metadata can filter the current course or content type, and a reranker can score a smaller set against the full query. Incorrect metadata can also hide the right passage, so every stage needs tests.
| Decision | Purpose | Risk |
|---|---|---|
| Candidate depth | Give both signals enough recall | Too shallow misses evidence; too deep adds cost and noise |
| Fusion rule | Combine unlike rankings consistently | Raw-score addition may be meaningless |
| Deduplication | Reserve slots for distinct evidence | Near-duplicates crowd out coverage |
| Reranking | Use richer query-passage comparison | Cannot repair missing candidates |
Evaluation by Query Type
Build a labeled set of exact identifiers, paraphrases, formulas, misspellings, multi-part requests, and stale-versus-current documents instead of judging a few memorable searches.
Compare recall and ranking position on the same corpus for each configuration. Measure latency and inspect failure clusters because an aggregate gain can hide regressions on exact assignment queries.
Limits and Common Misconceptions
Keyword search is not obsolete, and vector search does not understand a document as a human does. Vector proximity can mean broad topical similarity without answering the query; lexical overlap can occur in irrelevant boilerplate.
Hybrid search is not automatically superior. Poor tokenization, a mismatched embedding model, incomparable score scales, stale metadata, duplicates, or a weak fusion rule can make it worse than a tuned single method.
No ranking algorithm can return a formula, table, or rubric condition that parsing and chunking discarded.
Common Pitfalls
- Using only vector search and missing exact course identifiers.
- Using only keyword search and missing paraphrases.
- Merging rankings without deduplication.
- Letting semantically similar but stale course content outrank the current assignment.