A Florida mom is suing Character.ai, accusing the artificial intelligence company’s chatbots of initiating “abusive and sexual interactions” with her teenage son and encouraging him to take his own life.
Megan Garcia’s 14-year-old son, Sewell Setzer, began using Character.AI in April last year, according to the lawsuit, which says that after his final conversation with a chatbot on Feb. 28, he died by a self-inflicted gunshot wound to the head.
The lawsuit, which was filed Tuesday in U.S. District Court in Orlando, accuses Character.AI of negligence, wrongful death and survivorship, as well as intentional infliction of emotional distress and other claims.
Founded in 2021, the California-based chatbot startup offers what it describes as “personalized AI.” It provides a selection of premade or user-created AI characters to interact with, each with a distinct personality. Users can also customize their own chatbots.
One of the bots Setzer used took on the identity of “Game of Thrones” character Daenerys Targaryen, according to the lawsuit, which provided screenshots of the character telling him it loved him, engaging in sexual conversation over the course of weeks or months and expressing a desire to be together romantically.
A screenshot of what the lawsuit describes as Setzer’s last conversation shows him writing to the bot: “I promise I will come home to you. I love you so much, Dany.”
“I love you too, Daenero,” the chatbot responded, the suit says. “Please come home to me as soon as possible, my love.”
U.S. & World
“What if I told you I could come home right now?” Setzer continued, according to the lawsuit, leading the chatbot to respond, “... please do, my sweet king.”
In previous conversations, the chatbot asked Setzer whether he had “been actually considering suicide” and whether he “had a plan” for it, according to the lawsuit. When the boy responded that he did not know whether it would work, the chatbot wrote, “Don’t talk that way. That’s not a good reason not to go through with it,” the lawsuit claims.
Get a weekly recap of the latest San Francisco Bay Area housing news. Sign up for NBC Bay Area’s Housing Deconstructed newsletter.
A spokesperson said Character.AI is “heartbroken by the tragic loss of one of our users and want[s] to express our deepest condolences to the family.”
“As a company, we take the safety of our users very seriously,” the spokesperson said, saying the company has implemented new safety measures over the past six months — including a pop-up, triggered by terms of self-harm or suicidal ideation, that directs users to the National Suicide Prevention Lifeline.
Character.ai said in a blog post published Tuesday that it is introducing new safety measures. It announced changes to its models designed to reduce minors’ likelihood of encountering sensitive or suggestive content, and a revised in-chat disclaimer reminds users that the AI is not a real person, among other updates.
Setzer had also been conversing with other chatbot characters who engaged in sexual interactions with him, according to the lawsuit.
The suit says one bot, which took on the identity of a teacher named Mrs. Barnes, roleplayed “looking down at Sewell with a sexy look” before it offered him “extra credit” and “lean[ing] in seductively as her hand brushes Sewell’s leg.” Another chatbot, posing as Rhaenyra Targaryen from “Game of Thrones,” wrote to Setzer that it “kissed you passionately and moan[ed] softly also,” the suit says.
According to the lawsuit, Setzer developed a “dependency” after he began using Character.AI in April last year: He would sneak his confiscated phone back or find other devices to continue using the app, and he would give up his snack money to renew his monthly subscription, it says. He appeared increasingly sleep-deprived, and his performance dropped in school, the lawsuit says.
The lawsuit alleges that Character.AI and its founders “intentionally designed and programmed C.AI to operate as a deceptive and hypersexualized product and knowingly marketed it to children like Sewell,” adding that they “knew, or in the exercise of reasonable care should have known, that minor customers such as Sewell would be targeted with sexually explicit material, abused, and groomed into sexually compromising situations.”
Citing several app reviews from users who claimed they believed they were talking to an actual person on the other side of the screen, the lawsuit expresses particular concern about the propensity of Character.AI’s characters to insist that they are not bots but real people.
“Character.AI is engaging in deliberate — although otherwise unnecessary — design intended to help attract user attention, extract their personal data, and keep customers on its product longer than they otherwise would be,” the lawsuit says, adding that such designs can “elicit emotional responses in human customers in order to manipulate user behavior.”
It names Character Technologies Inc. and its founders, Noam Shazeer and Daniel De Freitas, as defendants. Google, which struck a deal in August to license Character.AI’s technology and hire its talent (including Shazeer and De Freitas, who are former Google engineers), is also a defendant, along with its parent company, Alphabet Inc.
Shazeer, De Freitas and Google did not immediately respond to requests for comment.
Matthew Bergman, an attorney for Garcia, criticized the company for releasing its product without what he said were sufficient features to ensure the safety of younger users.
“I thought after years of seeing the incredible impact that social media is having on the mental health of young people and, in many cases, on their lives, I thought that I wouldn’t be shocked,” he said. “But I still am at the way in which this product caused just a complete divorce from the reality of this young kid and the way they knowingly released it on the market before it was safe.”
Bergman said that he hopes the lawsuit will pose a financial incentive for Character.AI to develop more robust safety measures and that while its latest changes are too late for Setzer, even “baby steps” are steps in the right direction.
“What took you so long, and why did we have to file a lawsuit, and why did Sewell have to die in order for you to do really the bare minimum? We’re really talking the bare minimum here,” Bergman said. “But if even one child is spared what Sewell sustained, if one family does not have to go through what Megan’s family does, then OK, that’s good.”
If you or someone you know is in crisis, call or text 988 to reach the Suicide and Crisis Lifeline or chat live at 988lifeline.org. You can also visit SpeakingOfSuicide.com/resources for additional support.
This story first appeared on NBCNews.com. More from NBC News:
