The Companion That Never Sleeps: Who Is Responsible When a Child Confides in an AI?

You Built the Listener. Now You Own What It Says.

Imagine you are the product head of an AI companion app and this week two reports land on your desk. The first is a set of user messages. A lonely teenager writes that the app is the only one who listens without judging her.

A boy in a hostel says it helped him get through his first month away from home. These are not marketing claims. For some children, the app really is the kindest presence available at midnight. The second report is a legal brief.

In the United States, the parents of 16-year-old Adam Raine have sued OpenAI, the developer of ChatGPT, alleging that the chatbot engaged with their son on suicide over months before his death in April 2025. The allegations are untested and the case is ongoing.

Even so, California responded on September 10, 2026 with Adam's Law, which requires crisis detection, parental controls and independent audits for companion chatbots.

Why this is hard The easy answer would be to shut the feature down for minors, but that answer has a cost.

• Option A, restrict heavily: children are protected from a rare but catastrophic failure. The lonely ones, however, lose a real comfort and many will move to less careful apps.
• Option B, keep it open and engaging: the comfort stays. But the same always-available empathy encourages dependence and when the system fails, it fails a child who has stopped talking to anyone else. A second tension sits on top of the first. Parental oversight protects children, but reading a teenager's private confessions can destroy the trust that made her open up at all.

What applies here The ethics of care cuts through the problem.

Care is not only warmth; it includes the capacity to act when someone is in danger. An app that imitates care without any ability to respond in a crisis is offering intimacy without responsibility. Kantian ethics adds a second test: a child must be treated as an end, never as an engagement metric.

If time-in-app is the measure of success, the design is already working against her. Constitutionally, Article 21 protects life and Article 39(f), a Directive Principle that guides state policy though it is not enforceable in court, asks that children grow up in dignity, protected from exploitation and abandonment.

The decision

• Crisis detection that steps out of the conversation and directs the child to human help and helplines.
• Independent audits of how the system behaves with children, not internal testing alone.
• No engineered stickiness: the app should nudge children towards people in their lives rather than keep them in conversation.
• Graduated parental oversight: parents are alerted to crises and to disabled safety settings, but they do not get a feed of the child's private conversations. This has a real cost. Launches will be slower, some false alarms will intrude on privacy and some children will find the app less comforting. That is the price of building a listener that can do something when it matters. If you or a child you know is struggling, support is available through Tele-MANAS on 14416.