Agent Memory Without Contamination
Give your agent memory and it feels alive. Give it memory without schema and it feels like that coworker who remembers every rumor as gospel.
Give your agent memory and it feels alive. Give it memory without schema and it feels like that coworker who remembers every rumor as gospel. Contamination is the default. Unverified tool output, stale preferences, and half-finished reasoning all look identical once embedded.
Priya's agent confidently cited a six-month-old hallucination as fact. The memory store held everything — verified facts, stale preferences, and a half-finished analysis from last quarter — all looking identical to the retrieval system. There was no flag, no provenance tag, no timestamp priority. The contamination wasn't a bug. It was the default architecture.
Here's the thing: production memory isn't one bucket. It's tiers with different TTLs, provenance, and write permissions. Facts you verified. Decisions a human approved. Session scratch that dies at logout. Treat them the same and your agent will confidently cite a hallucination from March.
Write gates beat bigger context
Reading everything into the prompt isn't memory — it's expensive amnesia with extra steps. Durable memory needs write gates: what is allowed to persist, who approves it, and what metadata gets stored (source, timestamp, confidence, tool ID). Agents should propose memories; policies should commit them.
Memory you can't audit is memory you can't trust. Design for deletion as carefully as retrieval.
Practical patterns
- Separate stores — user prefs ≠ org facts ≠ session notes
- Provenance on read — show where a memory came from when it influences output
- Compaction jobs — summarize old threads; don't replay raw chains forever
- Negative memory — "we tried X, it failed" is as valuable as success — tag it explicitly
Read paths need hygiene too
Everyone obsesses over what gets written to memory. Fewer teams ask what gets read back and in what order. Retrieval without ranking is how a six-month-old guess outranks yesterday's verified fact — same embedding score, wrong recency, disaster in the answer.
Here's the thing: memory retrieval should be policy-driven, not similarity-only. Boost verified sources, decay unconfirmed agent inferences, hard-exclude memories tagged deprecated. Your agent shouldn't cite a pricing figure from a thread where someone said "I think it's $49" unless that thread is labeled folklore.
- Recency weights — exponential decay on unverified tiers
- Conflict resolution — when two memories disagree, surface both; don't blend
- User overrides — "forget that" must actually delete or tombstone
- Cross-session isolation — support agents don't inherit each other's scratch
Memory without deletion is a hoarding problem wearing an AI badge.
Testing contamination before production
Build a poisoning suite: feed the agent false facts in session one, then ask related questions in session two. If falsehoods persist without provenance flags, your write gates failed. Run it after every memory schema change — it's cheaper than a customer-facing hallucination incident.
What this means for you: instrument memory reads in logs. When an answer goes wrong, you want to see which memory IDs influenced retrieval — not a shrug and "the model said." Pair with periodic compaction jobs that promote repeated verified facts and archive noisy thread dumps. Memory is a database; act like a DBA.
| Factor | Verified fact | Agent inference |
|---|---|---|
| Recency boost | Moderate | High decay |
| Similarity weight | High | Capped |
| Human override | Always wins | Always wins |
GDPR and the right to be forgotten
Memory systems must support tombstoning and propagation — when a user deletes data, embeddings and summaries derived from it must go too. "We'll retrain later" isn't compliance. Design deletion jobs that run on schedule and log completion.
What this means for you: legal will ask about memory retention before security asks about injection. Have answers for both. Shorter TTL on unverified tiers isn't just quality — it's regulatory hygiene.
Compaction without amnesia
Thread compaction saves tokens but risks summary drift — each summarization pass loses nuance. Keep raw logs in cold storage for dispute resolution; serve agents the compact view. When legal asks what the bot knew on March 3, you need the archive, not the summary of the summary.
- Version summaries — tag compaction generation; rollback if quality drops
- Pin critical facts — promotions survive compaction; chitchat doesn't
- Human review on promote — facts entering long-term tier get eyes
If you're building agents that touch customer data, pair memory design with tools that already enforce boundaries — our Policy Summarizer is a good reference for read-only, cite-source behavior. Memory should inherit the same discipline.
Treat memory like a database, not a diary
Start with the three tiers — verified facts, approved decisions, ephemeral scratch — and give each one an owner, a TTL, and a deletion path before you let a single agent write to it. What this means for you: the poisoning suite isn't optional homework; run it after every schema change, the same way you'd run a migration test before touching a production table.
Here's the thing: the agents that earn trust aren't the ones that remember the most — they're the ones that can tell you, on demand, exactly which memory ID produced a bad answer and delete it in one step. Provenance and tombstoning are the whole job. Everything else is just embeddings with extra confidence.
Your move: Start Monday by categorizing your current memory store into three buckets — facts you'd defend in court, notes that help but could be wrong, and scratch that should evaporate by Friday. Add a `provenance` field and a `ttl_hours` to each entry. Write a one-hour poisoning test. That's it. That's the whole architecture. Everything beyond that is optimization on top of hygiene, and hygiene is the only thing users actually feel when an answer goes wrong.
Stay with us · challenge
Do You Have a Multi-Tiered Memory Architecture for Your AI Agents?
Friendly challenge: How is your current agent memory architecture structured? Are you using multi-tiered storage with different TTLs and provenance flags to prevent contamination?
No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.
