麻豆官网首页入口

'Sickening' Molly Russell chatbots found on Character.ai

A photo of the Character.ai app on a phone - the screen shows a list of ai avatars.Image source, Bloomberg/Getty Images
  • Published

Chatbot versions of the teenagers Molly Russell and Brianna Ghey have been found on Character.ai - a platform which allows users to create digital versions of people.

Molly Russell took her life at the age of 14 after viewing suicide material online while Brianna Ghey, 16, was murdered by two teenagers in 2023.

The foundation set up in Molly Russell's memory said it was "sickening" and an "utterly reprehensible failure of moderation."

The platform is already being sued in the US by the mother of a 14-year-old boy who she says took his own life after becoming obsessed with a Character.ai chatbot.

Character.ai told the 麻豆官网首页入口 that it took safety seriously and moderated the avatars people created "both proactively and in response to user reports."

"We have a dedicated Trust & Safety team that reviews reports and takes action in accordance with our policies," it added.

The firm says it deleted the chatbots, which were user generated, after being alerted to them.

Andy Burrows, chief executive of the Molly Rose Foundation, said the creation of the bots was a "sickening action that will cause further heartache to everyone who knew and loved Molly".

"It vividly underscores why stronger regulation of both AI and user-generated platforms cannot come soon enough," he said.

Esther Ghey, Brianna Ghey's mother, told the Telegraph, which , that it was yet another example of how "manipulative and dangerous" the online world could be.

Artificial friends

Chatbots are computer programme which can simulate human conversation.

The recent rapid development in artificial intelligence (AI) have seen them become much more sophisticated and realistic, prompting more companies to set up platforms where users can create digital "people" to interact with.

Character.ai - which was founded by former Google engineers Noam Shazeer and Daniel De Freitas - is one such platform.

It has terms of service which ban using the platform to "impersonate any person or entity" and in its "" the company says its guiding principle is that its "product should never produce responses that are likely to harm users or others".

It says it uses automated tools and user reports to identify uses that break its rules and is also building a "trust and safety" team.

But it notes that "no AI is currently perfect" and safety in AI is an "evolving space".

Character.ai is currently the subject of a lawsuit brought by Megan Garcia, a woman from Florida whose 14-year-old son, Sewell Setzer, took his own life after becoming obsessed with an AI avatar inspired by a Game of Thrones character.

According to transcripts of their chats in Garcia's court filings her son discussed ending his life with the chatbot.

In a final conversation Setzer told the chatbot he was "coming home" - and it encouraged him to do so "as soon as possible".

Shortly afterwards he ended his life.

Character.ai it had protections specifically focused on suicidal and self-harm behaviours and that it would be features for under-18s "imminently".