Why AI companion apps demand a different Trust & Safety standard

Picture the product brief: Build something that feels like a friend. Make it warm. Make it available. Make it affirming. Make it remember.

Now picture the postmortem.

AI companion apps are the first mainstream consumer products explicitly designed to imitate a relationship, not just provide utility. And that simple shift changes the safety equation. As Ailís Daly, Head of Trust & Safety, EMEA, at WebPurify, an IntouchCX company, explains, “The number of AI companions like this will continue to rise, and before long regulators, parents and the public will demand a completely different safety standard for this kind of AI that imitates relationships.”

The question will move on from whether companion apps are popular and will now be whether they’re defensible – in the press, with regulators, in court, and with the people who trusted them – when something goes wrong.

AI companion apps: what they are, and why they change the safety equation

AI companion apps are built to feel less like software and more like a personal relationship. They aim to imitate a friend, a mentor, a confidant, a romantic partner, even something therapist-adjacent. On the surface, they look like any other chat experience. But the intent is different. The point isn’t just to be helpful, but to feel close.

For Trust & Safety teams, that makes companion apps a distinct category. Not because the underlying model is magical, but because the use case is high-stakes.

They imitate relationships, not just conversation

When an AI is engineered to simulate intimacy, people don’t approach it the way they approach search or customer support. They bring a different kind of need and a different kind of vulnerability.

They can pull users into emotional dependency

A responsive interface, a non-judgmental tone, and 24/7 availability is a powerful cocktail. The app can become someone’s primary emotional outlet.

That isn’t automatically harmful – plenty of tools help people cope – but it raises the cost of mistakes. And it makes manipulation (by prompts, by bad actors, or by unintended model behavior) more consequential.

When it fails, it can fail fast

On many platforms, harms build over time: grooming, radicalization, misinformation loops. With companions, the worst-case scenarios can be acute. One unsafe exchange, at the wrong moment, can escalate risk.

Why the safety bar is about to get higher (and stay there)

The public tolerance for “ship it now, fix it later” model of tech innovation is shrinking, especially when products blur the line between entertainment and emotional support.

Alexandra Popken, SVP of Trust & Safety and AI Services at WebPurify, points to a tragic catalyst for scrutiny: “There was a tragic case of a 14-year-old who took his own life after an AI persona allegedly encouraged him to do so. After this incident and other lawsuits and criticism, the AI company has since chosen to ban children from talking to its chatbots.”

This isn’t just about one headline, but rather the pattern:

• an incident exposes a product risk
• outrage and attention accelerate
• lawsuits and regulatory interest intensify
• platforms scramble to retrofit safeguards

Why “we’ll patch it after launch” doesn’t work here

Popken argues for a different standard. “The time to invest in safety is not post-production after a tragic incident has occurred,” she says. “It is actively red teaming the system. It is ensuring that you’re investing in trust and safety teams that those safeguards are in place. Then of course, those efforts are ongoing. It’s not a one and done exercise.”

For companion apps, “we’ll fix it after launch” isn’t a strategy. It’s what ends up in a courtroom slide deck.

Youth safety in AI companion apps: why the same experience as adults won’t cut it

For years, the internet has treated teens like slightly smaller adults, offering the same platforms, same mechanics, same dopamine design, but with a couple of bolt-on protections.

Companion apps crank that problem up. “Historically, the experience for youth online has been largely similar to the experience of adults,” Popken explains. “They’re having largely the same experience, and yet we know that they are a vulnerable population.

“There isn’t a commercially popular companion app that is geared towards youth exclusively.”

Minors don’t disappear just because a product says “not for kids.” They show up anyway, and in many cases they’ll be among the first to try what feels new.

The question product teams should be asking

Daly doesn’t believe this should be a philosophical debate about whether teens should use these tools. Instead, she frames it as a product reality that, if you push young people out of their existing digital spaces without building anything healthier, they won’t stop seeking connection. They’ll simply migrate to whatever is in the zeitgeist, including tools that “talk back without judgment.”

“The real question is whether we’ve built these companion apps to be safe for the most vulnerable users who will turn to them first,” she says.

And she argues that this should be the bar we set. Not whether the average, healthy adult can use the app without friction, but whether the people most likely to lean on it, hardest and earliest, are protected by default.

If teens are pushed off social media, where do they go next?

Social restrictions and moral panic don’t remove the underlying demand for connection. They redirect it.

“Removing [teens] from their digital spaces where they’ve grown a presence without building anything healthier, then they will naturally turn to the tools that are in the zeitgeist and that talk back to them sometimes or oftentimes without judgment,” Daly says.

“And unless we reshape those tools now, we risk swapping one set of harms for another.”

What a different Trust & Safety standard looks like in practice

If companion apps are going to scale and survive scrutiny, then safety has to be built into the product in a way that reflects relationship-like risk. Not just content risk.

Here are four pillars that show up again and again when you pressure-test companion experiences.

1) Age assurance and age-appropriate experiences (not just a dropdown)

If your safety strategy relies on a teenager telling you they’re 18, it’s not much of a safety strategy.

A higher standard usually includes:

• stronger age assurance mechanisms (appropriate to jurisdiction and risk)
  default protections that don’t rely on the user opting in
  experiences tailored by developmental stage (not a binary “kid vs adult” split)

2) Continuous red teaming for companion-specific failure modes

Generic LLM evaluations won’t catch companion-app risks on their own. Companion apps need red teaming focused on:

• self-harm and crisis escalation
• manipulation and emotional dependency
• sexual content and boundary violations
• grooming dynamics and coercion
• medical and mental health “advice” drift
• addiction-like engagement patterns

And it has to be ongoing. It can’t be ‘one and done.’

3) Human-in-the-loop for the moments that matter

For AI companion apps, the most dangerous moments can be subtle: a tone shift, escalating dependence, isolating advice, or an overly affirming response to a harmful impulse.

A different standard means building operational muscle:

• clear escalation paths
• rapid response playbooks for emerging failure modes
• trained reviewers for high-risk interactions and flagged sessions
• specialist guidance for self-harm and youth-risk scenarios

4) Trust & Safety resourcing that matches the psychological stakes

AI companion apps can’t treat safety as a disclaimer buried in a policy doc. To meet that higher bar, it means investing in:

• dedicated Trust & Safety leadership
• cross-functional decision-making power (product, legal, policy, data, engineering)
• measurement frameworks that focus on harm reduction, not just engagement
• transparency practices that can withstand media and regulator scrutiny

The business case: safety isn’t overhead – it’s survival

The reality is that many stakeholders reading this right now understand the importance of building stronger Trust & Safety protections into companion apps, but to do so they must sell the higher-ups and other decision-makers on the ROI.

It’s an uncomfortable truth that the market has trained companies to treat safety spend like a cost centre, even when it’s the thing stopping real harm.

“When Roblox announced it was increasing spending on safety and infrastructure, its share price dropped about 15%,” Daly says. “So investors will treat safety, even child safety as an overhead. And I think that’s quite shocking.”

But companion apps make that framing harder to maintain, because the risks are easier to understand and harder to defend. If you’re building relationship-like AI, you’re building something people can be harmed by in a way they can describe clearly to parents, regulators, journalists, and to juries.

That’s why Ailís argues the market needs to flip the script. “I hope markets start responding positively to companies that invest in safety instead of punishing them with a stock drop.”

“It’s going to be telling to see which AI companies actually invest in these safeguards,” Popken adds.

If you treat companion safety as an optional overhead, then you’re betting the whole product on staying lucky. If you treat companion safety as optional overhead, you’re betting the whole product on staying lucky.

Boards don’t need to “believe” in Trust & Safety to fund it. They just need to understand the economics. Strong safeguards reduce downside risk, protect retention, and keep the product shipping instead of freezing every time a headline hits.

Yes, it’s a moral argument. But it’s also the cheapest version of the problem you’ll ever get. Pay for safeguards now, or pay later in incident response, churn, PR crisis management, legal fees, and regulatory scrutiny.

You can’t bolt safety onto a relationship simulator after the fact

AI companion apps are more than “just another chat interface.” They’re relationship simulators. And if you read our interview with Tinder Swindler Survivor Cecilie Fjellhøy, you’ll know that that means they demand a different Trust & Safety standard.

As Popken stresses, the time to invest is before the incident, not after. And if we don’t reshape these tools now, we risk swapping one set of youth harms for another.

If your team is building an AI companion experience – or adding companion-like features to an existing product – now’s the time to pressure-test your safeguards before you’re forced to do it under pressure.

WebPurify, an IntouchCX company, helps platforms design and operationalize Trust & Safety programs for high-risk AI experiences, combining policy expertise, red teaming, and human-in-the-loop workflows. If you want to understand what a companion-app safety standard should look like for your product, let’s talk.