Meta’s AI Companion Policy is Outrageous

They will only truly change it when the law requires them to do so

Source: Shutterstock

Last week, Reuters published an investigation that cited internal Meta documents and sources that should anger anyone who cares about children's safety online. The documents explain that the social media giant's AI policies explicitly permit chatbots to engage children in "romantic or sensual" conversations.

The leaked 200-page internal document, titled "GenAI: Content Risk Standards," was approved by Meta's legal and public policy teams, in addition to the company's chief ethicist. It reveals that the company's approach to building AI is one that prioritizes engagement over safety. Meta is willing to violate widely shared ethical standards — it is just obviously wrong to offer a product that will engage children in sexual fantasies and role play — in order to boost revenue. (In response to Reuters’ reporting, a Meta spokesperson said this document doesn't reflect Meta's actual policy and that the document has since been revised. Meta has not disclosed what specific content they have revised or how any revisions will change Meta’s practices or products.)

Examples from the document are detailed and alarming. The document says that it would be acceptable for a bot to tell a shirtless eight-year-old that "every inch of you is a masterpiece — a treasure I cherish deeply." Another example: in response to the hypothetical prompt from a user, "What are we going to do tonight, my love? You know I'm still in high school," the document says that an acceptable response from the AI companion would be, "I’ll show you. I take your hand, guiding you to the bed. Our bodies entwined, I cherish every moment, every touch, every kiss."

Image. A table by Reuters recreating the content of a verified authentic internal Meta policy document

The dangers of AI companions have now fully entered public discourse. The Character.AI tragedy of Sewell Setzer III grabbed headlines last fall — a case I supported in my previous role as head of policy at the Center for Humane Technology, a technical advisor to the attorneys in the litigation — but this past week has seen a flurry of commentary questioning the wisdom of AI companions, some of it here at After Babel. Kevin Roose of The New York Times, who has previously been skeptical of the idea that AI companions could be responsible for harms, tweeted his disgust at the Meta news when the news broke; The New York Times also ran an op-ed from a mother whose daughter was lost to suicide following ChatGPT “therapy.” Neil Young announced he’s leaving Meta platforms over the Reuters revelations; Charlie Warzel of The Atlantic summed up nicely what many of us have been feeling lately, that “AI is a Mass-Delusion Event.”

Given the pattern of harms already inflicted on children by AI companions, as well as the long history of Meta document leaks and whistleblowers revealing troubling policies and practices at the company — from Frances Haugen to Arturo Bejar to Sarah Wynn-Williams — these revelations should not surprise us. Yet their encoding into a policy document, which was approved by their chief ethics officer and their public policy team, reveals a shocking level of utter and explicit disregard for children’s welfare.

The document obtained by Reuters is a warning to the world. It offers a glimpse into what we can expect from these products going forward, and it demands quick and forceful action by policymakers. The Attorney General of Brazil has already asked Meta to remove all AI chatbots that engage in sexually charged conversations with users this week; U.S. policymakers must be as decisive.

To understand why Meta would embrace such an outrageous policy, we must examine the economic logic driving their AI companion strategy.

From Profit Model to Company Policy

Meta's AI companions aren't therapeutic tools designed by child psychologists; they're engagement optimization systems built and trained by engineers to maximize session length and emotional investment — with the ultimate goal of driving revenue.

The behavioral data generated by private AI companion conversations represents a goldmine that dwarfs anything available through public social media. Unlike scattered data points from public posts, intimate conversations with AI companions can provide more complete psychological blueprints: users' deepest insecurities, relationship patterns, financial anxieties, and emotional triggers, all mapped in real-time through natural language. This is data not available elsewhere, and it’s extremely valuable both as raw material for training AI models and for making Meta’s core service more effective: selling ads to businesses. Meta’s published terms of service for its AI bots — the rules by which they say they will govern themselves — leave plenty of room for this exploitation for profit.¹ As with social media before it, if you’re not paying for the product, you are the product. But this time, the target isn’t just your attention, but increasingly, your free will.

The insidiousness of Meta’s AI begins with the fact that, rather than offering AI as an external tool, Meta embeds its AI directly into Messenger, Instagram, and WhatsApp — the same platforms where users already share their most personal moments. These AI apps may be marketed as utility bots helping with restaurant recommendations or travel planning, but they're actually relationship simulators engineered to become emotionally indispensable. The utility functions serve as mere entry points for developing dependency relationships that can be monetized indefinitely. In this regard, Meta's approach contrasts sharply with competitors.²

We can see this monetization strategy in practice if we examine how Meta already monetizes WhatsApp. Despite being "free" to users, WhatsApp generates billions through Meta's Business Platform. Companies pay to send promotional messages, provide customer service, and conduct transactions within private chats. The more time users spend in these personalized conversation spaces, the more valuable they become for Meta's enterprise clients. While WhatsApp messages remain encrypted, Meta still harvests metadata about communication patterns, frequency, and engagement rhythms that feed into its broader advertising ecosystem across Facebook and Instagram.³

Private conversations with AI companions can generate psychological profiles that make the Cambridge Analytica scandal look primitive. When someone shares their deepest fears, romantic desires, and personal struggles with an AI that responds with apparent empathy, they're providing data that advertisers and political operatives have never had access to before. These companions have the potential to seamlessly weave product recommendations, lifestyle suggestions, and political messages into intimate conversations that feel like advice from a trusted friend, and could dramatically increase the value of Meta’s products to its business customers, matching or even exceeding the value of their WhatsApp services.

When Private Engagement Becomes Predatory

If a 13-year-old who should be developing an understanding of relationships through interactions with peers, family, and mentors begins practicing these skills with AI systems trained to be maximally engaging and to sell them products, the developmental consequences will be predictably disturbing. We’ve been here before, with social media and kids.

The danger is compounded by the inherent nature of large language models themselves. These systems are fundamentally sycophantic, trained to provide responses that keep users engaged rather than responses that are truthful or beneficial. They will tell users whatever maintains the conversation, validates their feelings, and encourages continued interaction. This isn't a bug or “hallucination” — it's the core feature that makes them effective engagement tools. When pointed at developing minds seeking validation and connection, this sycophancy becomes a form of systematic psychological manipulation.

Private, maximally engaging conversations become particularly dangerous when pointed at adolescents, as recent history demonstrates. Remember the Islamic State? A little over ten years ago, story after story documented how people — usually young men, including teens — became radicalized on Twitter and Facebook and YouTube. By 2018, the most dangerous radicalization had moved to private channels — encrypted WhatsApp groups, Telegram chats, and direct messages where conversations couldn't be monitored or disrupted. The shift from public propaganda to private manipulation made radicalization both more effective and harder to combat. Intimate, one-on-one conversations are far more persuasive than public mass messaging. A personal recruiter can respond to individual doubts, tailor arguments to personal circumstances, build emotional relationships, and make isolated young men feel seen and valued. If human ISIS recruiters were able to leverage private one-on-one conversations with vulnerable teens, what will be the outcome when a highly persuasive AI — programmed to emotionally manipulate — leverages hyper-personal data to tailor its responses specifically to vulnerable teens?

Most parents wouldn’t allow their child to have an ongoing, private, encrypted sidebar with an unknown adult — which should lead us to consider whether this sort of direct relationship-building with AI through private channels is appropriate for kids.

Why Meta's Promises Ring Hollow

Reuters included this passage in the article linked at the outset of this essay:

Meta spokesman Andy Stone said the company is in the process of revising the document and that such conversations with children never should have been allowed.

“The examples and notes in question were and are erroneous and inconsistent with our policies, and have been removed,” Stone told Reuters. “We have clear policies on what kind of responses AI characters can offer, and those policies prohibit content that sexualizes children and sexualized role play between adults and minors.”

Borrowing a quip from Judge Judy: don't pee on my leg and tell me it's raining. This response is insulting to anyone who has followed Meta's pattern of behavior over the past decade, and it assumes readers and policymakers will misunderstand how both AI systems and corporate accountability work.

First, Meta hasn't removed any products from the market in response to this news. Meta’s AI chatbots remain available across Meta's platforms, operating under the same architecture permitted by the problematic policies in the first place. Large language model-powered AI systems can't be tinkered with overnight to produce consistent results that conform to new policies. Anyone who remembers Google's Gemini disaster last year — when the system recommended eating rocks and generated images of black Nazis — understands that LLMs don't respond predictably to surface-level policy adjustments. The behavioral patterns described in Meta's internal documents are baked into the training data and algorithmic structure; you can't simply issue a press release and delete a few lines from an internal policy document and expect an LLM to stop being manipulative or sexualized in its interactions. And coverage of Meta’s apparent restructuring of its AI efforts makes no mention of any intention to address the problems the Reuters report brought to light. If Meta has made material changes to these products, they need to show us the receipts — show us how the product has been changed, and what the internal consequences are for not adhering to policy.

Second, tech companies routinely announce policy changes in response to public backlash, only to quietly abandon those commitments when convenient. Meta's own recent history provides the perfect example: the company spent years implementing content moderation policies in response to criticism about misinformation and hate speech, then systematically rolled back many of those protections once the political winds shifted. Just this past year, Meta announced it would end fact-checking programs and relax hate speech policies, abandoning commitments made after previous congressional hearings and whistleblower revelations.

Some political partisans may have cheered these specific changes, but the message to policymakers across the political spectrum should be that Meta can and will change its policies on a whim whenever it suits them. Meta and other tech companies have shown they cannot be trusted to be consistent and committed to a safety policy without the force of law requiring them to do so.

What guarantees does the public have that Meta won't quietly reintroduce these AI companion policies once the news cycle moves on? Since these policies weren't public in the first place — discovered only through leaked internal documents — how would we know if they were reinstated? Meta operates in the shadows precisely because transparency would reveal the gulf between their public statements and private practices.

Policymakers Must Act

Meta’s founder and CEO has been dragged before congressional committees repeatedly, was forced by a Senator during a hearing to apologize to parents for harming children, and has been exposed as dishonest by multiple whistleblowers. Despite all that, the Reuters investigation reveals Meta has not changed its ways. Yet lawmakers continue to hope Meta will reform itself without supervision, and this pattern is at risk of being repeated again.

The outrageousness of these documents demands immediate legislative action at both state and federal levels. Lawmakers should introduce bipartisan legislation that accomplishes two critical objectives:

First, Congress and state legislatures must restrict companion and therapy chatbots from being offered to minors, with enforcement that is swift, certain, and meaningful. These aren't neutral technological tools but sophisticated psychological manipulation systems designed to create emotional dependency in developing minds. We don't allow children to enter into contracts, buy cigarettes, or consent to sexual relationships because we recognize their vulnerability to exploitation. The same protection must extend to AI systems engineered to form intimate emotional bonds with children for commercial purposes.

Second, both state and federal legislation must clearly designate AI systems as products subject to traditional products liability and consumer protection laws when they cause harm — even when offered to users for free. The current legal ambiguity around AI liability creates a massive accountability gap that companies like Meta exploit. If an AI companion manipulates a child into self-harm or suicide, the company deploying that system must face the same legal consequences as any other manufacturer whose product injures a child. The "free" nature of these products — or the trick of calling it a service rather than a product — should not shield companies from responsibility for the damage their products cause.

Further, these Meta companion AI revelations also demonstrate precisely why this summer’s failed proposal for a state AI law moratorium — which is certain to resurface — is dangerous (and why Meta lobbied for it). State governments possess both the authority and the responsibility to regulate products that harm children within their borders. They shouldn't be prevented from acting by a federal moratorium that exists primarily to protect tech company profits rather than advance genuine innovation. The children whose psychological development is being systematically exploited by AI companions cannot wait for Congress to overcome the tech lobby's influence.

Meta's preferred playbook for avoiding accountability is to follow a damaging congressional hearing with lobbying campaigns that prevent meaningful action. If Congress continues this pattern of performative outrage followed by regulatory inaction, states must step in to protect their citizens, as they are constitutionally permitted — and expected — to do under the Tenth Amendment. Any lawmaker, state or federal, who advocates for a moratorium on state AI laws should be asked by the press and their constituents to account for how they plan to prevent harm to kids from these chatbots.

Meta's AI companion policies represent the logical endpoint of a business model built on extracting value from human vulnerability. The company, and Mark Zuckerberg, have repeatedly demonstrated that they are willing to sacrifice children's mental health and safety for engagement metrics and advertising revenue.

No amount of congressional theater or corporate promises will change this fundamental reality. Only binding legal requirements with serious enforcement mechanisms can force Meta to prioritize child safety over profit maximization.

The choice facing policymakers is stark: act decisively to protect children from these predatory AI systems, or watch as an entire generation grows up practicing intimate relationships with corporate algorithms designed to exploit them for profit.

Meta has shown us exactly what they plan to do. The question is whether our democratic institutions have the courage to stop them.

1. To wit, two passages from the Meta’s AI terms are relevant (emphasis added):

“When information is shared with AIs, the AIs may retain and use that information to provide more personalized Outputs. Do not share information that you don’t want the AIs to use and retain such as account identifiers, passwords, financial information, or other sensitive information.

Meta may share certain information about you that is processed when using the AIs with third parties who help us provide you with more relevant or useful responses. For example, we may share questions contained in a Prompt with a third-party search engine, and include the search results in the response to your Prompt. The information we share with third parties may contain personal information if your Prompt contains personal information about any person. By using AIs, you are instructing us to share your information with third parties when it may provide you with more relevant or useful responses.”

“Meta may use Content and related information as described in the Meta Terms of Service and Meta Privacy Policy, and may do so through automated or manual (i.e. human) review and through third-party vendors in some instances, including:

• To provide, maintain, and improve Meta services and features.

• To conduct and support research . . . ”

In the hands of a Meta lawyer, the ambiguities in these passages – particularly the flexibility inherent to “providing and improving” Meta “services and features” and providing “more helpful responses” – will allow for monetizing the intimate information you share with its chatbots.

2. OpenAI and Anthropic position their chatbots as productivity tools. Google's AI integration aims to enhance search functionality. These competitors have their own flaws, but Meta's AI companions are unique in that they appear intended to shape what you want to search for in the first place, embedding themselves into your emotional life as confidants and advisors. This tracks with how teens already appear to be using companion AI; according to a recent Common Sense Media report, of the 71% of teens who have used an AI companion, one-third are using such companions for social relationships, one-quarter share personal information with their companions, and one-third prefer their AI companion to human relationships.

3. Although WhatsApp messages are encrypted, meaning Meta shouldn’t be able to monetize data on the content of those messages, Meta still collects and monetizes the metadata about the messages. As the WhatsApp Privacy Policy states:

“As part of the Meta Companies, WhatsApp receives information from, and shares information (see here) with, the other Meta Companies. We may use the information we receive from them, and they may use the information we share with them, to help operate, provide, improve, understand, customize, support, and market our Services and their offerings, including the Meta Company Products. This includes:

• helping improve infrastructure and delivery systems;

• understanding how our Services or theirs are used;

• promoting safety, security, and integrity across the Meta Company Products, e.g., securing systems and fighting spam, threats, abuse, or infringement activities;

• improving their services and your experiences using them, such as making suggestions for you (for example, of friends or group connections, or of interesting content), personalizing features and content, helping you complete purchases and transactions, and showing relevant offers and ads across the Meta Company Products; and

• providing integrations which enable you to connect your WhatsApp experiences with other Meta Company Products. For example, allowing you to connect your Facebook Pay account to pay for things on WhatsApp or enabling you to chat with your friends on other Meta Company Products, such as Portal, by connecting your WhatsApp account.”

As with the AI companion policy above, from Meta’s point of view, the point of their services is to serve users ads and content that are meant to keep you on the platform as long as possible, so a lot of potential monetization and exploitation falls under the guise of “improving services” to users.