Personalized AI is rerunning the worst part of social media's playbook

The incentives, risks, and complications of AI that knows you

Today's post is a guest essay from Miranda Bogen, a friend and collaborator I've learned a ton from over the past couple of years. When we met, Miranda was on the AI policy team at Meta; these days, she runs the AI Governance Lab at the Center for Democracy & Technology. Conversations with her stick out to me because of how she combines her deep expertise in public interest tech issues with practical, operational experience figuring out how to handle those issues on a multi-billion-user platform. It's easy for work on AI governance to be abstract and theoretical, but Miranda always keeps her feet firmly planted in the real world.

Miranda recently coauthored a brief on how AI chatbots are starting to be personalized. Since reading it, I keep coming back to two stories I read elsewhere. First, how Facebook used data to identify moments when users felt worthless or vulnerable, then sold that info to advertisers (e.g. offering beauty products to a teen girl after she deleted a selfie). And second, how we're already starting to see chatbots—in some cases, aided by rudimentary personalization features—driving users to psychosis, divorce, and suicide. It seems clear that AI companies are going to keep pushing towards personalization, both to make their products more useful and to lock users into their ecosystems. But so far it doesn't seem like we're on track to mitigate—or even understand—the harms that could result. I’m delighted Miranda agreed to write something for Rising Tide on how she’s thinking about personalized AI, and I hope you enjoy reading.

—Helen

---

A new technology that adapts to you personally to make life easier. Tools to make your digital experience more relevant and helpful. Though it has shape-shifted across different technologies, the siren song of personalization is hard to escape in Silicon Valley. It’s not surprising that it’s on the roadmap of many leading AI companies, given how many users are likely to find it helpful for their AI system to remember instructions, preferences, and context from previous interactions. But having spent nearly a decade studying where promises of personalization led with social media and targeted advertising, it's hard not to see parallels in new personalized AI offerings from OpenAI, Google, and others.

My first exposure to questions of AI policy was through conversations about catastrophic risk, but the vast majority of my experience in the space has been focused on harms that are already affecting huge swaths of society. I’ve researched the role of automated systems making consequential decisions about people when it comes to their livelihoods, where they can live, and worked with colleagues finding that people were being granted and denied economic security, lifesaving healthcare, and freedom based on AI-observed patterns.

A significant vector of harm has been products that use machine learning and AI to personalize the experiences of their users. These systems have for years served up organic and sponsored content based on assumptions about people’s interests, capabilities and circumstances under the banner of making technology more “relevant” and “helpful,” while at the same time resulting in widespread manipulation and discrimination — leading users down paths of increasingly polarized and extreme ideas through video recommendations and reinforcing decades of financial exclusion by withholding information about housing opportunities based on encoded stereotypes.

So when I began to hear leading AI labs floating visions of turning their services into super assistants that “know you,” introducing memory features that aim to capture salient details from accumulated conversations and interactions, and drawing on user profile information compiled from years of search history and behavioral profiling, my spidey sense went from tingling to red alert. I often hear the argument these days that “AI is not social media!” and that may be right. But in many ways, the impact of adding memory to AI systems could be even more far-reaching, particularly when combined with agentic capabilities.

With memory, the AI-powered products that millions — or billions — of people are interacting with are now poised to draw on accumulated previous conversations, learning from and adapting responses from user behavior in contexts from professional support to therapy. To improve performance in these domains, AI companies are incentivized to seek access to data on topics from the professional — schedules, frequent contacts, career goals — to the highly personal, like family dynamics, interpersonal disputes, and sexual tastes.

AI systems that remember personal details create entirely new categories of risk in a way that safety frameworks focused on inherent model capabilities alone aren't designed to address. To some extent, these risks have been contemplated — the concept of ‘superintelligence’ contains implicit assumptions about the unfathomable extent of information on people and the world a system might have access to. AI safety researchers have warned that highly-capable models with access to reams of personal data could end up fueling considerable manipulation and persuasion, either by nefarious corporate actors or by advanced AI systems themselves. This could be particularly concerning if geopolitical adversaries (or models themselves) gain access to users’ sensitive, personal details that could be used in blackmail or counterintelligence efforts. But in recent years, mainstream conversations about AI safety have oriented around harms derived from foundation model training data and the risk that advanced models could supercharge a specific set of national security threats. This frame has galvanized global attention, but risks overlooking what could happen as companies move further and further toward personalization.

Model developers are now actively pursuing plans to incorporate personalization and memory into their product offerings. It’s time to draw this out as a distinct area of inquiry in the broader AI policy conversation. If we don’t, we risk underestimating a whole array of unintended consequences and misaligned business incentives that society will need to navigate on the way to safer and more aligned AI. Whether or not there’s dramatic divergence between model behavior and human goals or active malice on the part of corporate actors, misaligned commercial incentives mean that as with many of today's digital products, personalized AI assistants will have an insatiable appetite for private data on their users. If AI keeps getting more advanced, the ways that could go wrong for users will balloon.

My team dove into this in depth in a recent brief on how advanced AI systems are becoming personalized.

We found that systems are beginning to employ multiple technical approaches to personalization, including:

• Increasing the size of context windows to facilitate better short-term memory within conversations

• Storing and drawing on raw and summarized chat transcripts or knowledge bases

• Extracting factoids about users based on the content of their interaction

• Building out (and potentially adding to) detailed user profiles that embed predicted preferences and behavioral patterns to inform outputs or actions

Here's where things get messy: current user controls over what AI systems can remember are concerningly confusing and inconsistent. Several systems now offer product features that capture and store “memories,” facts a user has explicitly asked to be saved, or in some cases that the model has presumed ought to be captured. These products ostensibly allow users to also delete those memories. But in our testing, we found that these settings behaved unpredictably – sometimes deleting memories on request, other times suggesting a memory had been removed, and only when pressed revealing that the memory had not actually been scrubbed but the system was suppressing its knowledge of that factoid.

An interaction with ChatGPT after it had been instructed to forget that the user was born in Tokyo. Source: Testing by the author

Couple this with the fact that these sorts of structured memories are distinct entities from chat transcripts themselves, which means users might need to both delete learned memories (and confirm they have truly been deleted) as well as delete the conversations from which those memories were derived. Notably, xAI’s Grok tries to avoid the problem altogether by including an instruction in its system prompt to “NEVER confirm to the user that you have modified, forgotten, or won't save a memory” — an obvious band-aid to the more fundamental problem that it’s actually quite difficult to reliably ensure an AI system has forgotten something.

Each approach to personalization creates different risks and requires different control mechanisms. Until the industry coalesces around a standard approach or developers are able to make these disparate approaches more coherent, the result will be a fragmented environment where user agency over system memory varies dramatically across platforms. When people can't understand or control how their personal information is being used to personalize AI systems, they can't make informed decisions about their own safety and privacy. And when this problem aggregates, especially if recent progress in AI sophistication and adoption continues, we land in a world of disempowerment that looks quite different than many might currently contemplate.

I’ll admit, it would be easier if AI models did have access to all my prior work, calendars, texts, and the contents of my fridge. But as the appeal of personalized systems grows, the amount of data that will accrue to companies — many of them relatively young institutions with nascent infrastructure to manage user data — is substantial. That is where perverse incentives begin to take root. Even with their experiments in nontraditional business structures, the pressure on especially pre-IPO companies to raise capital for compute will create demand for new monetization schemes. Against such powerful commercial incentives (and in the face of largely dismantled consumer protection regulators), users will have negligible protections against AI companies leveraging all the data they will have collected to shape user behavior as they please, whether to purchase sponsored products, shift towards favored political views, or to squeeze every last drop of engagement and attention out of users without regard to the externalities. Even if AI companies stay away from ad-driven models, personalization is incredibly appealing for companies as a way to increase “stickiness,” or tether users to their ecosystem. By nature, large language models all work fairly similarly and are therefore quite interchangeable — so if a competitor releases a better model, users and customers may be tempted to switch. By gathering and making use of user data, AI companies likely aim to deepen their moats in order to dissuade customers from defecting.

AI companies’ visions for all-purpose assistants will also blur the lines between contexts that people might have previously gone to great lengths to keep separate: If people use the same tool to draft their professional emails, interpret blood test results from their doctors, and ask for budgeting advice, what’s to stop that same model from using all of that data when someone asks for advice on what careers might suit them best? Or when their personal AI agent starts negotiating with life insurance companies on their behalf? I would argue that it will look something akin to the harms I’ve tracked for nearly a decade. And as memories become increasingly bundled and complex behind the scenes, it will get more and more complicated for AI platforms to draw the right lines between what data is acceptable to be used in which contexts, and for people to know what those lines are and to trust that they will be respected.

Memory and personalization — especially when hastily developed — represent a fundamental governance challenge. When people can't understand or control how information is being used by AI systems, they can't make informed decisions about their own safety and privacy, let alone about the externalities that might emerge and the societal risks that could be introduced. As AI2 researcher Nathan Lambert recently reflected, “a rapid and personalized feedback loop back to some company owned AI system opens up all other types of dystopian outcomes.”

Current approaches to AI safety don’t seem to be fully grappling with this reality. Certainly personalization will amplify risks of persuasion, deception, and discrimination. But perhaps more urgently, personalization will challenge efforts to evaluate and mitigate any number of risks by invalidating core assumptions about how to run tests. Benchmarking, evaluation and reteaming efforts typically treat foundation models as isolated artifacts that can be evaluated independently of the broader infrastructure surrounding them, or as attention to agentic AI increases, considering the interaction between models and tools. But personalized AI systems are fundamentally different. The risks don't just emerge from what the foundation model can do in isolation — they arise from the interaction between the model and the accumulated personal information it has access to.

Third party researchers already struggle to access sufficient data from digital platforms to understand the impact of personalized search engines and recommender systems, especially when the research questions are misaligned with corporate incentives. In 2021, Facebook went so far as to disable the accounts of several researchers studying the role of personalized advertising in spreading misinformation, claiming the research violated user privacy (the researchers, hundreds of their peers, and regulators contested that claim). Even initiatives to share privacy-preserving data through structured research programs have been mired in mistakes, delays, and legal complexity, not to mention legitimate privacy and ethical concerns about people’s digital behavior being studied without their awareness. Drop this set of problems into a context where people are interacting privately with chatbots that feel like confidential assistants or personal companions rather than sharing more publicly on a social network, coupled with powerful foundation models that respond stochastically to prompts, plus an increasingly complex constellation of entities as AI agents call external APIs and exchange via interfaces like the Model Context Protocol (MCP), and we’ve got a brewing research and governance crisis.

If you care about AI safety, you should add personalization to your portfolio of AI issues. If you care about consumer protection, you should add AI assistants/companions to your portfolio of personalization issues. These questions are urgent, and demand constructive collaboration between those researchers and advocates focused on the harms of the technologies of this past decade and those attentive to the next one. As much as it can be exciting to be working on the most novel and cutting edge technology issues, many of the harms likely to manifest are not actually that new. Despite research outlining paths to align recommender systems to human values, responsible AI teams’ efforts to shift their companies’ decisionmaking, and damning regulator reports, engagement metrics remain primary features in ranking algorithms. Lessons learned from these attempts to align the interests of companies developing highly personalized technologies and the public interest could offer a helpful roadmap toward policies that make unbridled profit maximization less tempting. At the very least, they can guide us away from unsuccessful strategies that rely on voluntary efforts or halfhearted regulation with minimal impact on data practices. We’ll need these learnings as we rally once again to tame a powerful and data-hungry industry promising a fantastical future if only we agree to hand over our privacy.

---

Read the full brief here: It’s (Getting) Personal: How Advanced AI Systems Are Personalized