Personalisation vs privacy: drawing the line deliberately

The creepiest AI feature we ever reviewed was built by nice people with good intentions. A health app drafted "insights" that began "Since your sleep dropped after your late shifts last week…" — accurate, helpful, and deeply unsettling to the nurses testing it, because none of them remembered telling the app it was watching their rotas. Technically they had: a permissions screen in onboarding. Practically, the product had crossed a line nobody had ever consciously drawn. The line existed; it was just discovered by a user, in a session recording, with her hand over her mouth.

That's the failure mode this article is about. Personalisation with AI now has a much longer reach — models infer things users never typed, remember across sessions, and can surface inferences in prose that feels eerily knowing. The old privacy playbook (a policy page, a cookie banner) doesn't cover it. You need to draw the line as a design act, before legal is asked to. Here's the framework we use on engagements and the patterns that keep personalisation on the right side of it.

The line is a product decision, documented like one

Every personalised feature has three properties that determine where it lands: what it reads (declared data, behavioural data, inferred data), what it remembers (session, device, server-side profile), and how visible the inference is in the output ("we recommend hiking boots" vs. "because you run at 5am in winter"). Creepiness is largely a function of the third: users tolerate what they can explain.

So before design starts, we run one workshop — designers, engineers, a legal or privacy advisor, and someone with the authority to say no — and fill in a one-page table: for each planned personalisation, the three properties above, the value it delivers, and the story you'd tell a user if they asked "how did you know that?" The last column is the whole point. If the honest answer to "how did you know that?" makes anyone in the room wince, the feature gets redesigned or cut. This is the same instinct we apply in our public responsible-AI review: the time to find the wince is in a meeting room, not in a usability lab.

Document the outcomes. Not in the privacy policy — in an internal register that names the decision, the date, and who made it. Privacy decisions age; the register is how you remember what "on purpose" meant two years later.

Pattern 1: Personalise from what's on the device, not who's on the wire

The strongest privacy architecture is the one where sensitive inference never leaves the user's hardware. Local models are now good enough for real personalisation jobs: ranking a user's own content, drafting text in their voice from their own documents, summarising their own notifications. When the data and the model both live on-device, you get features that feel intelligent with a privacy story that is trivially honest: "This runs on your phone. We never see it."

Even without local models, prefer what we call ephemeral personalisation: the inference happens from the session's context and is forgotten. Recommend from the current cart, the current document, the last five actions — personalised in effect, stateless in fact. It's remarkable how much of what teams call personalisation ("help me pick up where I left off") needs a session, not a dossier.

Server-side long-term profiles are the last resort, reserved for cases where the value genuinely compounds over months and is impossible locally. When you do build one, every field in it needs its own justification — a profile is a liability ledger, and each entry accrues risk.

Pattern 2: Make personalisation explain itself, cheaply

Users forgive being known when they can see the mechanism. The pattern that consistently works: a small, ever-present "Why this?" affordance next to any personalised output, answering with the actual input used — "Because you read three articles about composting" or "Based on your last order". Not a paragraph of legalese; one sentence, scannable in a glance, linked to controls.

This does two jobs. It converts surveillance-feeling into service-feeling ("of course, that's fair"). And it functions as a continuous audit: if product teams find their own "Why this?" answers embarrassing to display, the feature has already failed. We've killed two planned features this way, in design review, by just writing the honest "Why this?" copy and looking at it. Embarrassment is a privacy metric with a very fast feedback loop.

For AI features that cite or retrieve, the same principle extends further — provenance is the AI-era version of "Why this?" and deserves real interface design, which we detail in citation design for AI answers.

Pattern 3: Consent that means something

Most consent UX is trained-incapacity: a banner people click through to reach the product, which means it consents to nothing and informs no one. Meaningful consent for AI personalisation has four properties.

Scoped. Ask about the specific use, not "personalised experience". "Generate summaries of your health data to discuss with your doctor" is a scope. "Improve our services" is not.

Timed. Ask when the feature is first relevant, not at install. Consent requested in context converts better and means more, because the user understands what they're trading. This is the same lesson as permission UX: the ask lands when the value is visible.

Revocable in one place, actually. A single settings surface where each consent can be withdrawn, where withdrawal visibly changes behaviour ("we'll forget and stop doing X"), and where the system honours it — including purging the derived profile, not just the toggle. Test the purge. We've audited systems where "off" disabled the feature and retained the inference for a subsequent internal project. That's a breach of the design's own promise.

Refusable without penalty. The no-thanks path must be as easy as the yes and leave the product fully functional, minus the personalisation. Dark patterns in consent don't just erode trust — in several jurisdictions they make the consent legally void, which means you built the data asset on nothing.

Pattern 4: Retention on a schedule, deletion on a promise

Data that outlives its purpose converts from asset to liability on a timer. Our default schedules, offered to clients as a starting point: session context dies with the session; behavioural event streams aggregate or expire at 90 days; derived inferences refresh from scratch rather than accreting forever; account deletion means deletion, including backups on a declared window (30–90 days), stated plainly in the privacy policy.

The disciplines that make this real are unglamorous: every data store has an owner and an expiry in the schema, expiry is enforced by code that runs, and someone reviews the enforcement quarterly. If deletion is a manual runbook, it doesn't happen.

Pattern 5: The honest-capability copy doubles as privacy copy

The irony of this whole topic: the fix is mostly words. We've written elsewhere about honest capability statements — naming what a feature reads, what it remembers, and its boundaries. The same statements, written well, are your privacy interface. "Summarises this month's spending. Nothing leaves your account. History is forgotten after 90 days" is simultaneously great feature copy, great privacy notice, and a testable commitment the engineering must honour. Write it early and hold the build to it.

Key takeaways

• Draw the line in a workshop before design: data read, memory horizon, and visible inference — plus the "how did you know that?" column. Wince-inducing answers kill features early, which is the cheapest time to kill them.
• Prefer on-device and ephemeral personalisation; treat server-side long-term profiles as a liability ledger where every field needs justification.
• A one-sentence "Why this?" affordance on every personalised output is both a trust feature and a continuous self-audit.
• Consent must be scoped, timed to relevance, revocable with real purging, and refusable without penalty.
• Retention schedules enforced by code, not runbooks; deletion includes derived data, on a stated timeline.

FAQ

Isn't personalisation expected now? Do users actually care?

They care about the feeling, not the feature. Personalisation that feels like service ("based on your last order") is welcomed; personalisation that feels like surveillance ("we noticed you sleep badly") gets screenshotted and shared as a horror story. The design work is keeping everything in the first category.

We already have a privacy policy. Isn't that enough?

A policy is where the record of your commitments lives, but users never read it — the interface is the real privacy notice. Per-feature statements, "Why this?" affordances and honest consent flows are where the understanding actually forms.

Does meaningful consent tank opt-in rates?

In our experience, scoped and well-timed asks perform better than blanket early banners, because users understand the trade. If explaining a feature honestly makes users refuse it, that refusal is information about the feature, not a UX problem.

Who should own this inside a company?

Product, with legal as advisor and veto-holder — never the reverse. When the line is drawn by legal alone, you get compliance-shaped products that users experience as cold. It's a design decision with legal constraints, like an accessibility standard.