source: arxiv artificial intelligence: deployment-time memorization in foundation-model agents
level: research
foundation-model agents that remember users over time face a trade-off between helpful personalization and privacy. researchers frame this as a deployment-time memorization problem, where memory design choices directly control what the agent recalls and what an attacker can extract. they introduce two metrics: personalization recall (pr) for utility and adversarial extraction rate (aer) for risk. a third metric, forgetting residue score (frs), checks if deleted data still lingers in derived memory layers.
the team tested three design knobs on the longmemeval benchmark: how aggressively summaries compress facts, how many past items the agent retrieves (k), and how deletion is handled. key-fact summarization cut canary extraction by 76% on gemma 3 compared to full-text storage. reducing retrieval breadth also lowered extraction but hurt personalization. deletion mode mattered too—soft deletion left more residue than hard deletion, meaning supposedly forgotten data could sometimes be recovered from summaries or embeddings.
the results show no single setting optimizes both privacy and utility. aggressive summarization helps privacy but can degrade recall. narrow retrieval protects data but makes the agent less helpful. the forgetting residue score reveals that even after deletion, information can persist in compressed forms. the work provides a framework for developers to navigate this frontier, choosing memory configurations based on their specific privacy needs and performance goals.
why it matters: it gives ai developers measurable ways to balance personalization and privacy in long-lived agents, directly impacting user trust and regulatory compliance.
source: arxiv artificial intelligence: deployment-time memorization in foundation-model agents