Memory is a UX surface: designing what assistants remember

An AI assistant that remembers nothing is a search box with manners. An AI assistant that remembers everything is a surveillance product wearing a friendly hat. Between those poles is the design space that actually matters: a memory system the user can see, edit, and trust. We treat memory not as a model capability but as a UX surface — as designed, inspectable and argued-over as any settings screen. Here's how we design it, after shipping memory features across three client products and shelving one entirely (the right call, more below).

Why memory is different from every other feature

Memory inverts the usual privacy-beneath-the-floorboards arrangement. Telemetry and cookies are invisible by default and users tolerate them precisely because they're invisible. Assistant memory is invisible and conversational — it surfaces unprompted, in a context the user didn't choose. "Since you mentioned you're gluten-free…" is a feature. It is also a reminder, delivered mid-task, that the system builds a model of you. When that reminder lands well, it feels like being known. When it lands badly — wrong fact, wrong moment, wrong intimacy — it feels like being watched, and "watched" is unrecoverable in a way "forgettable" never is.

That's why the design order is reversed from how teams usually build it. Don't ask "what can we remember?" Ask "what would the user be glad we remembered, and under what conditions would they want to know?"

The memory taxonomy

Not all memories are equal, and pretending they are produces creepy assistants. We design in four tiers:

Session context — what's in the current conversation. Free, expected, forgotten when it ends. No design burden beyond a visible context indicator.
Explicit preferences — things the user stated as instructions: "Answer in German", "Always show prices with tax". Store durably, show in a memory list, honour everywhere. These are settings the user happened to type.
Inferred facts — things the system deduced: dietary restrictions from past orders, working hours from activity, expertise level from vocabulary. The risky tier. Requires the interface patterns below.
Identity-adjacent data — health, finance, family, beliefs. Default rule: do not infer, do not store unless explicitly saved by the user, and treat with stricter retention than everything else. Some products should refuse this tier entirely.

The taxonomy drives everything downstream: what appears in the memory UI, what gets used silently versus announced, and what expires.

The memory screen is the product

Every memory feature we ship includes a memory surface — a place where everything remembered is listed, in plain language, with edit and delete. Not buried in settings; linked from every point where memory is used. "Why do you know that?" should always have an answer one tap away: because this, stored then, shown here.

Design details that earn their place:

Plain-language rendering. Not a JSON blob of extracted entities — a sentence a person would write in a notebook: "Prefers morning meetings (added 14 May, from your calendar patterns)." Each entry carries provenance: when it was learned and from what. Provenance is what separates "the assistant noticed" from "the assistant is spying", as with citations in trust design generally.
Inline confirmation for inferred facts. When the system first wants to keep an inference, it asks, in context: "Want me to remember that you're usually booking for two?" One tap to accept, one to decline. Confirmed inferences convert from creepy to considerate; silently stored ones never do.
Edit, not just delete. Memory will be wrong. "I'm vegetarian now" is an edit, not an erasure, and a good memory UI lets people correct the record the way they'd correct a colleague's misunderstanding. The correction affordance principle carries over directly from designing AI failure states.

Forgetting on purpose

The feature nobody builds is forgetting, and it's the one that makes memory tolerable. Three kinds:

Expiry. Inferred facts decay. Last winter's project is not a permanent resident of the assistant's world model. Time-stamp everything, expire inferred facts on a schedule (we default to 90 days of no reinforcement), and tell users the policy in one sentence: "I forget things you don't repeat."

Context walls. What the assistant knows about your personal account shouldn't leak into your employer's workspace. Memory scopes must match the user's mental model of context, not the database's foreign keys. Cross-context bleed is the single most damaging memory failure we've seen in testing — users forgive a wrong fact far sooner than a fact surfacing in the wrong room.

Ritual deletion. A visible "forget this conversation" and, for products that earn it, timed auto-forget as a marketable posture. Deletion must be deletion — the record and its embeddings, with a user-visible confirmation. "I've forgotten our conversation" backed by a soft-delete flag is a lie waiting for a security researcher.

The privacy postures that earn retention

Memory features live or die on uptake and retention: what fraction of users leave memory on after month one. The postures that move the number:

Opt-in for the sensitive, default-on for the trivial. Preferences: on by default, loudly disclosed. Identity-adjacent inferences: off until asked.
Disclosure at use, not at consent. The first time memory changes an output, say so: "I remembered you prefer metric units." Consent dialogs are forgotten in a day; in-context disclosure teaches the feature continuously.
Local mental models. People understand "my assistant remembers" through the metaphor of a person. Use it honestly: a discreet human assistant tells you what they've noted when it matters and never mentions what they haven't. Build the same etiquette.

This is where memory intersects product governance: a memory taxonomy, retention schedule and disclosure policy are exactly the artefacts our responsible-AI review audits, and they're easier to get right at design time than to retrofit after the first uncomfortable support ticket.

When not to build memory

We talked one client out of it: an HR tool whose assistant could have remembered performance-review context across conversations. The value was real. So was the blast radius — an imperfect, inferred, editable-but-mostly-not record of sensitive workplace judgments, sitting between an employee and their manager. The product already had a trust problem to solve; memory would have multiplied it. We shipped session memory and excellent export instead. The test we now apply: if a stored memory showed up verbatim in a user's data-export request, would the user nod or flinch? Ship what makes them nod.

For the flows around deleting and exporting, the patterns in cancellation flows that leave the door open transfer surprisingly well — offboarding memory is offboarding, and the same respect applies.

Key takeaways

Design memory as a UX surface: a visible, editable list with provenance for every entry.
Classify memories into tiers — session, stated, inferred, identity-adjacent — with different rules for storage, use, and disclosure.
Ask before keeping inferred facts; confirm in context. Silent inference is the creepiness generator.
Build forgetting deliberately: expiry, context walls, and real deletion with visible confirmation.
Earn retention with disclosure at the moment of use, not with a consent dialog on day one.
Some products shouldn't have memory. Apply the data-export test before you build.

FAQ

Isn't this what "memory on/off" toggle solves?

No. A global toggle outsources an architecture decision to a user who can't see the trade-offs. Most users want some things remembered and others never mentioned again. The toggle is the floor; the tiered taxonomy and the memory surface are the feature.

Should memory improvements be visible in evals and testing?

Yes — memory behaviour belongs in the eval suite like any other behaviour: does the assistant use stored preferences correctly, refuse to leak across contexts, and forget on schedule? Unmeasured memory drifts as fast as any prompt does, especially through model migrations.

What about "the user can always look at their data" regulations?

Design to the spirit rather than the letter: the export should be legible to a human, memory should be individually deletable, and consent for sensitive tiers should be meaningful (specific, in-context, reversible). If your memory UI is genuinely inspectable, compliance follows from the design rather than the other way round.

How do enterprise customers react to assistant memory?

Enterprises buy memory they can govern: admin visibility over scopes, workspace-level retention policies, and guarantees against cross-tenant contamination. Consumer memory ethics and enterprise memory governance are the same design at different scales — provenance, boundaries, and honest deletion.

Can memory replace good onboarding?

No — they compound. A good assistant onboarding flow teaches the memory model explicitly: what it keeps, where it lives, how to prune it. Teach it at the start and the memory screen reads as a feature; skip it and the first discovered memory reads as a disclosure.