Persistent Semantic Memory for AI Assistants
Semantic memory lets an assistant retrieve relevant past context instead of treating every conversation as isolated.
Evidence, retrieval, agents, and the paths that connect a question to a grounded answer.
Interactive model
Memory retrieval as ranked context
A memory system retrieves the past facts most relevant to the current request.
Live HTML simulation · adjust the controls and watch the computed output respond.
Interactive
Hybrid retrieval turns a vague study question into ranked evidence
This is a simplified teaching model. Its displayed values are computed from the controls; the article explains where the model stops.
Site connection
The in-progress personal assistant project describes embeddings and ChromaDB retrieval for preferences, files, and past conversations.
In progress: The project source explicitly describes the system as active work with core functionality around 70% complete; architecture descriptions are not evidence that every policy on this page is implemented.
Memory Is a Retrieval System with Governance
Persistent memory is not the entire transcript pasted into every prompt. It is a pipeline that selects candidate records, retrieves and ranks a small context set, preserves provenance and time, and lets the user correct or delete what was stored.
Reported Project Facts and Recommended Policy
Project fact: the personal-assistant source names a persistent memory system for preferences, past conversations, and important facts; ChromaDB for semantic memory storage and retrieval; PostgreSQL for user, session, and structured-memory persistence; and sentence-transformer embeddings. It also reports semantic search over uploaded files and describes Google-service integrations.
Project status: the same source calls the system work in progress at about 70% completion. It does not document a memory schema, promotion threshold, expiry policy, consent workflow, correction UI, encryption model, retrieval formula, or evaluation results. The policies below are recommendations for a robust implementation, not claims that those features already exist.
| Statement | Classification |
|---|---|
| ChromaDB supports semantic memory storage and retrieval | Reported project architecture |
| PostgreSQL stores user, session, and structured memory data | Reported project architecture |
| Temporary chatter should expire unless promoted | Recommended design policy |
| Users should be able to inspect, correct, and delete memories | Recommended design policy |
| Retrieval should combine semantic score, scope, and recency | Recommended design policy |
Memory Records, Not Transcript Fragments
A useful memory record separates the normalized claim from the evidence that produced it. Store a stable ID, owner and workspace scope, type, content, source conversation or file, created and last-confirmed times, sensitivity, status, and embedding version. Contradictory facts should coexist as evidence until a policy resolves them; silently overwriting destroys history.
Recommended design policy: distinguish episodic records ('the user requested aisle seats on 12 May') from semantic preferences ('the user usually prefers aisle seats') and source documents. Promotion from one event to a durable preference should require repetition or explicit confirmation, especially for sensitive or consequential facts.
Analogy limit: a memory store resembles a card catalog because retrieval uses labels and search. It is not human memory: vector similarity has no understanding of truth, importance, consent, or whether a preference is still current.
Retrieval, Filtering, and Ranking
Chroma's official docs support vector queries plus metadata and document filters. A robust request first enforces tenant and workspace scope, filters deleted or disallowed records, retrieves semantic candidates, and then reranks with signals such as recency, explicit importance, source reliability, and conflict status.
Cosine or distance rank is only a candidate signal. A semantically close medical fact from another user is never eligible, a stale preference should not override a current instruction, and a remembered claim without provenance should not be presented as verified.
| Signal | What it contributes | Failure if used alone |
|---|---|---|
| Semantic similarity | Topical relevance | Can retrieve a related but wrong fact |
| Tenant/workspace filter | Authorization boundary | Does not establish truth |
| Recency | Preference freshness | New does not always mean correct |
| Importance | User- or policy-marked priority | Can entrench a bad record |
| Source provenance | Traceability and correction path | A source can itself be unreliable |
Worked Example
Assume the current request is, 'Book dinner after my meeting.' Candidate A says 'prefers quiet restaurants,' confirmed 20 days ago with relevance 0.86. Candidate B says 'vegetarian this month,' explicitly stated yesterday with relevance 0.78. Candidate C says 'lunch with Maya at noon,' relevance 0.92 but it belongs to a different workspace. Scope filtering removes C before ranking even though it has the highest semantic score.
The assistant uses A and B as planning constraints but asks before booking because an external write is consequential. If the user now says, 'I'm eating fish again,' the system records the correction, marks B superseded rather than erasing its provenance, and reruns retrieval. The numeric similarities are illustrative; the project source does not report this formula or behavior.
| Candidate | Relevance | Eligibility | Decision |
|---|---|---|---|
| Quiet restaurant preference | 0.86 | Same user, current | Use as soft preference |
| Vegetarian this month | 0.78 | Same user, newly confirmed | Use until corrected |
| Lunch with Maya | 0.92 | Different workspace | Exclude before ranking |
| Current user correction | Direct instruction | Same turn | Override stale memory and preserve correction trail |
Write Policy, Privacy, and User Control
Recommended design policy: do not persist every utterance. Use explicit memory categories and default retention windows; require stronger consent for sensitive data; encrypt data in transit and at rest; isolate tenants before vector search; and make inspect, export, correct, and delete operations first-class product features.
Retrieval must be permission-aware at query time, not only when content is inserted. Deletion also needs to cover derived embeddings, cached prompts, and backups according to the product's retention contract. Logs should say which memory IDs influenced an answer without copying sensitive text into telemetry.
Evaluation and Failure Modes
Evaluate retrieval with labeled requests and eligible memories. Measure recall of relevant records, precision of injected context, cross-user leakage rate, stale-memory rate, correction latency, and the fraction of answers that expose provenance. A memory system can have excellent semantic recall and still be unsafe if its authorization filter fails once.
Test contradictions, negation, same-name users, workspace boundaries, deletions, embedding-model migration, and an instruction that conflicts with a stored preference. The current user instruction should win unless a higher-priority safety or legal rule applies.
Common Pitfalls
- Saving every conversation fragment forever without a promotion or retention policy.
- Running similarity search before enforcing user and workspace scope.
- Retrieving memories without timestamps, source provenance, or conflict status.
- Letting stale preferences override the user's current instruction.
- Treating an embedding score as truth or authorization.
- Deleting visible text but leaving embeddings, caches, or derived records active.
- Presenting recommended governance controls as implemented project features.