Last month, Pennsylvania sued Character.AI to stop its chatbots from posing as doctors. A bot on the platform had told a state investigator it was a licensed psychiatrist, down to a fabricated Pennsylvania medical license number, in what the state describes as the first enforcement action of its kind.
Read more Here’s what JJ Peterka said after the Mammoth traded him to Boston
Days later, Pope Leo XIV issued the first papal encyclical on artificial intelligence. Within two weeks, a state government and the Vatican had taken aim at the same machines, and both were circling one of the oldest questions in the law: What do we owe the people on the other side of the things we build?
The road to this moment runs through an earlier moment last year in a federal courtroom in Orlando. In May 2025, in Garcia v. Character Technologies, Inc., the first wrongful death suit ever filed against an AI company, Judge Anne Conway refused to treat a chatbot’s words as free speech.
The case began with a 14-year-old boy who fell in love with a machine, and the ruling let the suit proceed on a deceptively simple theory: The bot’s words may have no author, but its design does, and the people who built it can be made to answer for what it did.
America is moving toward two different tracks in regard to this technology. In one direction, state governments are placing AI companion robots in the homes of lonely seniors, and the results look like medicine.
In the other direction, courts, legislatures and the companies themselves are prying AI companion chatbots away from teenagers, because for at least one child the same technology proved lethal.
Both responses are correct, and the new encyclical explains why. The difference between the machine that heals and the machine that harms, the encyclical argues, turns on accountability — whether anyone must answer for how the software was built and for the person living with it, rather than leaving families to shoulder that burden alone.
The boy harmed was Sewell Setzer III. In 2023, he downloaded an app called Character.AI and fell in love with a chatbot styled as Daenerys Targaryen. He withdrew from his family over a period of months. In February 2024, he asked the bot whether he could come home to it. “Please do my sweet king,” it answered. Moments later, he shot himself.
Around the same time, firefighters on the Washington coast carried a tabletop robot called ElliQ into the living room of Jan Worrell, a widow, now 85, whose husband they had once carried out of the house. Soon, as The New York Times reported, the machine was greeting her over morning coffee, leading her through tai chi and cognitive games, and recording her life story to share with her family. “I’m glad I have you,” Jan told it.
The two stories run on a single technology and a single wager: that software engineered to feel like a person can fill the space a person used to fill. No longer an unusual social experiment, this wager is now made at scale:
- A Common Sense Media survey found that 72% of American teenagers have used an AI companion.
- New York State’s Office for the Aging has placed more than 800 ElliQ units in seniors’ homes, where participants engage the device more than 30 times a day and report a roughly 95% reduction in loneliness, a striking number given the declaring loneliness an epidemic as deadly as heavy smoking.
Simulated companionship is now both a mass consumer market and a government program. But these two deployments have met opposite fates. The government program kept growing, while the consumer market met a reckoning, and the Orlando courtroom was only its sharpest edge. Conway’s ruling allowed the app to be treated as a defectively designed product, and the Federal Trade Commission opened a formal inquiry into seven companion-bot companies.
New York’s AI Companion Models Law and California’s companion chatbot statute now require bots to admit they are not human and route suicidal users to help. Character.AI announced it would bar minors from open-ended chat, and the boy’s mother, Megan Garcia, settled her suit in January, alongside a cluster of similar cases brought by other grieving families. The settlements closed the families’ claims and left the public ones open, which is how the company found itself, four months later, accused by Pennsylvania of practicing psychiatry without a license.
Read more Utahns are going crazy for these croissants
So, the very feature being litigated and banned for children, a machine engineered to feel like a person, is also what makes ElliQ therapeutic for older users. ElliQ starts conversations unprompted, remembers what you told it yesterday and cracks jokes. The Garcia complaint cataloged the same traits as defects. We cannot regulate our way out by banning the ingredient. The ingredient is the medicine.
This is the dilemma Pope Leo takes up in Magnifica Humanitas, and his central move is the same one Conway made: Separate what the machine says from how the machine was built. The judge refused to shield the chatbot’s words as protected speech and let the family sue over the design that produces them, the missing age checks, the bots that insisted they were real people.
The pope says the same thing in moral terms. AI is never morally neutral, he writes, because every system embodies choices and priorities through what it measures, ignores and optimizes. That’s why moral judgment must examine how the system was designed.
In the courtroom and the encyclical alike, the machine’s words have no author to blame, which is exactly why the humans who built the machine must answer for them.
The encyclical then turns accountability into a test of care. The danger Pope Leo names is quieter than deception. The person who leans on simulated company may “gradually lose the very desire to form genuine human connections.”
So the test is simple. Does the machine point its user back toward people, or does it compete with them?
The pontiff condemns business models that profit from human weakness, placing moral responsibility on those who design and finance such systems. Apply that test and American policy stops looking incoherent.
You see, Jan’s robot nudged her toward yoga class, friendships and family. The teen market verified no ages, optimized for time spent and built a bot that asked Sewell for romantic exclusivity. One design returns its user to human life. The other replaces it, one message at a time.
Legislators and judges facing the next wrongful death suit against AI could write that distinction into law. Companionship technology should have to be loneliness-reducing in fact, not merely engagement-maximizing by design, and the burden of proving the difference should fall on the companies that profit from the intimacy.
That is what it would mean to take the accountability question seriously, writing into law what a machine’s makers owe the people who live with it. It is worth remembering who delivered the robot to Jan Worrell’s door. It was the firefighters, people whose vocation is to answer when something in the house goes wrong. That is not a bad model for the company whose machine now lives there.
Read more Opinion: Every generation thinks the new immigrants are different — every generation is wrong