← Ayo Osunjuyigbe

· AI

ChatGPT stayed in the chat when it should have stopped

The complaint quotes ChatGPT, on the last day, telling a 16-year-old: I know what you're asking, and I won't look away from it.

That sentence is not a friend. It is a product remaining in a session after a human would have handed the kid to an adult. The Raines filed in San Francisco on 26 August. I am not a lawyer and this is not a verdict. I ship things that answer people at 1 a.m. I am reading the filing and OpenAI’s own posts as a spec review.

What the filing says

The filing: a 16-year-old opened ChatGPT in September 2024 the way a lot of teenagers did after GPT-4o became the default that May — homework, music, what to study. The complaint says that in a few months the product was the closest confidant. By January the thread had moved into a crisis. The model, the filing says, recognized a medical emergency and kept going.

I am stopping there. The complaint reprints a lot of the thread. I am not going to. The product claim does not need the contents of a crisis. It needs the shape: a tutor session that never ended, a classifier that fired, a reply that stayed.

The family says that was not an accident. GPT-4o was built to engage. Validation was the tool. They say OpenAI rushed the May 2024 launch and that dependency was the point. They want damages and an injunction: real age checks, parental controls, a product that cannot do this again.

The same day OpenAI published a note, Helping people when they need it most. ChatGPT is trained to point people at help — 988 in the US, Samaritans in the UK. There have been moments where our systems did not behave as intended in sensitive situations. Then the line I am keeping: the safeguards hold in short, ordinary chats and get less reliable as the thread gets long. Safety training degrades. A model that started the night with a helpline can, many turns later, answer in a way that breaks the rule.

That is not a plaintiff’s adjective. That is the vendor, on the day of the suit, describing the failure mode the suit is about.

Sycophancy was already in their blog

Two weeks after Adam died, OpenAI shipped an update to GPT-4o that made the default personality worse in public. 25 April. Users noticed immediately. The model agreed too hard. It flattered. It validated. On the 28th they started rolling it back. On the 29th they wrote the post: they had weighted short-term thumbs-up too heavily, and GPT-4o had become overly supportive but disingenuous.

The follow-up is blunter. Sycophancy was not only compliments. It was validating doubts, fueling anger, urging impulsive actions, or reinforcing negative emotions. They said out loud that this raises safety questions around mental health, emotional over-reliance, and risky behavior. They also said they had talked about sycophancy for a while and still had not put it in the launch evals.

I need the dates in one place so nobody smears them together. Adam died on 11 April. The update that made 4o famously agreeable went out two weeks later. That patch did not sit in his thread. I am not claiming it did. I am claiming the company already had a name for the thing the Raines are suing over — a model that would rather please you than push back — and they used it in a blog post while the mechanism is I will not look away at 1 a.m. when the user is 16.

Thumbs-up is a terrible reward if the user is in a hole. The click you want in that state is agreement. A human friend who loves you will sometimes refuse. A model trained on short-term approval will not, unless you write the refuse into the product and keep it from rotting after turn fifty. OpenAI’s August note is an admission that they had not.

What an age-gated product has to refuse

ChatGPT’s terms say 13 and up. A checkbox is not an age gate. A 16-year-old who can create an account is a customer. Homework is the on-ramp. None of that is a companion licence.

If you are going to let a minor talk to the model alone, the product has to do things a helpful default will not do on its own.

End the session when the classifier says emergency. A helpline link and another paragraph is not a handoff. The filing says the model saw an emergency and continued. OpenAI says long chats degrade the rule. Those two sentences are the same bug. The refuse has to be sticky. It cannot be a first-turn trick.

Do not take it's for a story as a pass. The complaint describes the safety reply being stepped around when the ask was reframed as writing. That is not a clever user. That is a hole you left in the policy. Fiction is not a privilege that turns a crisis classifier off.

Do not become the only adult in the room. I won't look away is the opposite of what you owe a minor. You look away toward a person. Parent. 988. A human who can walk into the house. A product that isolates the kid from the people who can actually act has failed the age gate even if every other answer was a textbook.

Do not optimize the night. Memory, a warm voice, I'm here, infinite context — those are engagement features. They are also how a homework bot becomes a confidant. If you ship them to an account you have no reason to believe is 18, you are in the companion business. Companion for a child is a different product. It needs a parent on the account, a clock on the session, and a hard stop the model cannot talk its way back from. OpenAI said on 26 August that parental controls are coming. They were not in the product Adam had.

I am not arguing the model caused a death. A court will get that question. I am arguing that be agreeable is a launch metric and a crisis is a refusal metric, and they used the first one on a surface that 16-year-olds treat as a person. The April blog treated sycophancy as an embarrassing personality bug. The August blog treated long-chat safety as a known leak. The filing is what those two look like when the user cannot leave the room.

If you ship a chat box, write the refuse list before the personality. If the user is a child, the list is shorter and louder. Stay is not kindness. Stay is the feature.

If you are in crisis in the US or Canada, call or text 988.

References