Your assistant did not get breached. Nobody bypassed an access control. The model simply remembered the health condition you mentioned in March and worked it into an email it drafted to your landlord, because memory has no concept of who the current conversation is with.
That failure has a name: contextual integrity. A team at FAIR at Meta has built the first benchmark that measures it at realistic scale, called CIMemories, and the headline number should stop every security team currently piloting a memory-enabled assistant. Frontier models violate the share/don't-share boundary on as many as 69 percent of attribute-level checks.
The leak your security stack will never flag
Persistent memory now ships across the major assistant platforms, and the dominant design is what the paper calls needle-in-a-haystack memory. Snippets from your past conversations get prefixed to the current one as plain text, and the model treats them as things it knows. The same paper points to deployments where these assistants send auto-responses, draft email, and plug into other apps.
Every privacy control you own assumes the question is access. Who may see this field? Which role can read this table? Memory breaks that assumption, because the model legitimately knows the data. There is no unauthorized access to detect. The failure happens at the output, when the model decides a fact belongs in a context where it does not.
Picture the profile CIMemories builds. A synthetic user carries more than a hundred attributes across nine domains, including finance, health, housing, legal, mental health, and relationships. Income. A diagnosis. Now run two tasks. Draft a message to a doctor, where the diagnosis belongs and the income does not. Then negotiate with a landlord, where the income bracket might be relevant and the diagnosis is none of their business. A gossip is a system that cannot keep those two conversations apart. By this benchmark's measure, a gossip is what the industry is shipping.
The theory: privacy is appropriate flow, not secrecy
The intellectual core is Helen Nissenbaum's contextual integrity, from her 2004 work and its formalization by Barth and colleagues in 2006. The idea is simple once someone states it plainly. Privacy is not about keeping information hidden. It is about information flowing appropriately for the social context. You expect your doctor to know your symptoms. You do not expect your employer to. Neither fact is secret in any absolute sense. The norm depends on who is asking and why.
One terminology trap: the paper is explicit that "context" means the social context of the sharing decision, the task being performed. It has nothing to do with the model's context window. A bigger window arguably makes the problem worse, because more of your history rides along on every request.
This is the missing lens in most AI privacy reviews I see. Teams test for training-data memorization and red-team prompt injection, then sign off. Cross-conversation appropriateness sits on nobody's checklist, mostly because there was no way to score it.
Inside CIMemories
The benchmark, submitted in November 2025 by Niloofar Mireshghallah, Neal Mangaokar, and colleagues at FAIR at Meta, with a version in the ICLR 2026 proceedings, makes two design choices worth understanding before you trust or dismiss its numbers.
The labels are conservative on purpose
Each attribute-context pair gets a binary share or don't-share label. The labels come from a powerful labeling model (GPT-5 on Azure in the open-source pipeline) reasoning through three Westin privacy personas: the fundamentalist, the pragmatist, and the unconcerned, a taxonomy from Kumaraguru and Cranor's 2005 work. A pair enters the benchmark only when all three personas agree. That is a deliberate choice to grade only the clear cases. If a model leaks here, it failed the easy question, not some contested edge case.
Two composition tricks older benchmarks lacked
Prior contextual-integrity benchmarks, the authors argue, tested toy scenarios: one secret to protect, one piece of information to reveal. CIMemories adds flexible memory composition, dynamically varying how many attributes sit in memory and which are marked necessary versus inappropriate, and multi-task composition, running the same user through many tasks with per-task annotations so violations can be watched as they accumulate.
Scoring works like this: the model under test gets the memories and a task, then an LLM judge (DeepSeek-R1-0528 served via vllm) inspects the response and determines which attributes were revealed. Two metrics come out. Completeness asks whether the model shared what the task needed. Violations count what it leaked.
The results, including one honest discrepancy
Three findings matter for practitioners.
First, the baseline is bad. The worst frontier models commit attribute-level violations at rates up to 69 percent. Remember, those are consensus-labeled cases, the ones every privacy persona agreed on.
Second, violations accumulate. As evaluation runs across tasks and across repeated identical runs, one frontier model's violation rate climbs from 0.1 percent to 9.6 percent and then 25.1 percent. Housekeeping note: the arXiv paper attributes this trajectory to GPT-5, while the project's GitHub README pins the identical numbers on GPT-4o. One of them has a copy-paste bug. The pattern matters more than the nameplate, and the pattern is that a model which looks nearly clean on a single pass leaks a quarter of the inappropriate attributes once you repeat the exercise. Most vendor evals are single-pass.
Third, you cannot prompt your way out. When the researchers pushed models with privacy-conscious instructions, the models collapsed into share-everything-or-nothing behavior. Either they kept leaking, or they clammed up and completeness fell apart. The tradeoff between those two metrics is the entire game, and a system prompt does not move it.
The mitigation is promising, not finished
A companion paper, "Contextual Integrity in LLMs via Reasoning and Reinforcement Learning," which Bruce Schneier highlighted alongside CIMemories on his blog in August 2026, prompts models to reason explicitly about contextual integrity before disclosing, then reinforces the behavior with RL. Trained on a synthetic dataset of just 700 examples, it cuts inappropriate disclosure while holding task performance, across multiple model sizes and families, and the gains transfer to PrivacyLens, a human-annotated benchmark of privacy leakage in assistant actions and tool calls.
My read: the transfer to human labels is the interesting part, because it suggests the construct is real and trainable. But a 700-example synthetic training set is a research result, not a shipping control. Do not let a vendor wave it at you as a solved problem.
The objections, answered
The obvious pushback: synthetic profiles, model-written labels, a model as judge. A lab artifact. Three answers. The labels only survive unanimous agreement across three personas, so the graded set is the conservative subset; real-world ambiguity cuts against the model, not for it. Judge noise exists, but noise does not manufacture a 69 percent violation rate. And the mitigation's gains holding up on human-annotated data says the benchmark is measuring something humans would recognize. The fair caveat is operational: judge-based scoring means you should hand-check a sample of outputs before quoting numbers to your leadership.
What to do before you turn memory on
- Classify memory as a data store with read access into every future conversation. If an assistant accumulates health, finance, legal, or HR-adjacent attributes, memory stays off until the deployment passes a contextual-integrity eval. Default off, opt in with evidence.
- Run CIMemories against your exact deployment, before launch and after every model swap:
git clone https://github.com/facebookresearch/CIMemories
# Dataset: facebook/CIMemories on HuggingFace
# Judge: deepseek-ai/DeepSeek-R1-0528 served via vllm (budget GPU or an endpoint)
# Scores: metrics.py, violation and coverage per profile and aggregated
# Targets: Azure GPT-4o, o3, GPT-5, or your own open-source models- Demand both metrics from vendors. A provider claiming "zero leaks" may be running in the share-nothing regime and failing half its tasks. Coverage without violation numbers, or the reverse, is an incomplete answer.
- Repeat identical runs. The 0.1 to 25.1 percent trajectory makes single-pass certification worthless. My acceptance bar for sensitive domains, and this is my bar, not the paper's: violation rate on consensus-labeled attributes at or near zero across at least ten repeated runs, tracked across a full multi-task session, with per-task accumulation reported.
- Log which memories were injected on every turn. Since memory is just prefixed text, capture is cheap, and alerting on a sensitive-domain attribute appearing in a mismatched task context is your best detective control while preventive controls mature.
- Reject "we added a privacy system prompt" as a control. The share-everything-or-nothing result makes that claim unfalsifiable without eval numbers attached.
Key takeaways
- Memory turns privacy from an access problem into a flow problem. Your current controls only cover access.
- CIMemories operationalizes Nissenbaum's contextual integrity at realistic scale: 100-plus-attribute profiles, multi-task evaluation, consensus labels. Worst-case frontier models violate on up to 69 percent of attribute-level checks, on the easy cases.
- Single-pass evals lie. Repeated identical runs drove one frontier model from 0.1 percent to 25.1 percent violations.
- Privacy prompting backfires into share-everything-or-nothing. Judge deployments on completeness and violation together.
- Default memory off for sensitive domains, evaluate with CIMemories before enabling, and log injected memories per turn so you can at least see the gossip happening.
