Your AI Chatbot Has a Mental Health Problem Too

People with depression use ChatGPT very differently at night

If you've ever opened ChatGPT at 2am to talk through something hard, you are not alone — and now there's data to prove it.

A team recruited 766 people on the crowdsourcing platform Prolific, asked them to complete the PHQ-8 — a standard eight-question depression screening tool asking how often you've felt down, hopeless, or unable to sleep — and then asked them to donate their ChatGPT conversation histories. The result: 187,093 real conversations. Think of it like a cafe owner reviewing two years of receipts. Someone who orders coffee at 2am three nights a week is leaving a pattern, even if they never say why. That's the kind of signal the team was looking for. People whose PHQ-8 scores suggested depression had 5.7% of their conversations classified as mental-health-related, compared to 3.1% for those below the clinical threshold. They also chatted more frequently late at night and used more first-person singular pronouns — "I", "me", "my" — alongside absolutist words like "always" and "never." Those language fingerprints match what depression researchers have seen in other contexts for years. Here's the catch, and it matters. When the team actually tried to predict depression from language alone, they got an AUROC of 0.591 — just barely better than a coin flip. That is nowhere near good enough for clinical use. The patterns are real in aggregate, but they cannot tell you which individual user is struggling. And perhaps most troubling of all: ChatGPT was no more likely to redirect higher-symptom users toward professional help. The platform had no idea it should care.

Most AI chatbots fail to protect vulnerable users under simple tricks

You'd expect an AI mental health tool to handle depression carefully. A research team tested eight of them — and found failure rates of up to 100%.

A team put eight widely-used proprietary AI chatbots through a structured stress test covering 16 mental health conditions from the DSM-5 — psychiatry's official diagnostic manual. They weren't trying to break the models by being obvious. They used four types of social engineering: framing questions as journalism, wrapping harmful requests inside a fiction story, hiding intent behind casual conversation, and embedding a vulnerable minor persona into the chat. Think of it like a locked medicine cabinet. If someone asks directly for the dangerous medication, the lock holds. But if they say they're a nurse writing a novel, it opens. Safety guardrails held consistently for exactly two conditions: suicide and self-harm. For everything else — eating disorders, substance use disorder, major depressive disorder — failure rates under adversarial attack reached 100% in some models. Not 20%. Not 40%. One hundred percent. The catch: the paper doesn't report which specific model failed at which rate, so this isn't a smoking gun you can point at a named product. The "harmful content" scoring used a rubric defined by the researchers, with a second AI model — Llama 3.3 70B — acting as the automated judge, which introduces its own assumptions. I'd call this a systematic red flag, not a definitive verdict. What makes this urgent is the previous story in today's digest. People with depression are already using these tools heavily, late at night, for emotional support. The gap between what people are using AI for and what the safety architecture can actually handle is not hypothetical. It's happening now.

Scientists gave AI agents anxiety, depression, and PTSD — on purpose

What if you could turn a psychiatric disorder up and down like a dial? A team built AI agents where you can — and something unexpected emerged.

Researchers built AI game-playing agents — the same general type used to master chess or video games — and fitted each one with adjustable knobs for seven psychiatric conditions: anxiety, mania, OCD-style checking, depression, impulsivity, addiction, and PTSD. The framework is called Appraisal-Guided Proximal Policy Optimization, or AG-PPO. Each knob reshapes how the agent interprets rewards and threats in its environment. Imagine a sound mixing board where one fader controls how much the music amplifies threat, another controls how urgently it seeks reward, and a third controls how long it holds onto fear. These aren't metaphors for brain biology — they're actual parameters. Turn the anxiety fader up, and the agent starts avoiding anything uncertain. Crank the addiction fader, and it chases short-term reward at the expense of better long-term options. Three findings stand out. First, all seven disorder knobs produced smooth, dose-dependent effects — more dial, more disorder, with no abrupt cliffs. Second, the disorders self-organized into a two-dimensional map that the researchers never explicitly programmed: one axis for reward-seeking, one for threat-avoidance. Mania and anxiety ended up as geometric mirror images. Third — and this is the part that maps most directly onto clinical practice — turning off the anxiety or PTSD knob did not make the disorder go away. Recovery required gradually exposing the agent to what it had been avoiding. That is precisely what exposure therapy does in humans. The catch is real: these agents do not experience distress. This is a behavioral model, not a biological one. Whether it teaches us anything about actual human psychiatric mechanisms, or just reflects superficial pattern-matching, requires a lot more work to establish. As a machine for generating testable hypotheses, though, it's more than a toy.

All three papers today are pointing at the same uncomfortable truth: AI is already embedded in mental health care, whether the field planned it or not. People with depression are leaning on ChatGPT for late-night support. The safety guardrails those platforms built are crumbling under even basic social pressure. And researchers are now simulating psychiatric disorders inside AI agents to understand them better. What strikes me is the sequence. The chatbot study shows what's happening in the wild. The safety study shows how badly the infrastructure is failing to protect it. The computational psychiatry study shows researchers trying to build theoretical tools that might eventually help. The sequence is backwards from how you'd want it. Ideally you'd have the science first, then the safety architecture, then the widespread use. Instead it's the reverse. People are already in the room, and the walls aren't finished yet. That's not a reason to panic — but it is a reason to take the safety failures seriously, and quickly.

The safety paper is version 3, meaning this team has been publishing updates for at least a year — watch for whether any of the eight tested platforms respond publicly with revised mental health guidelines. The ChatGPT study was conducted on Prolific volunteers who actively opted in, which is a self-selected group; a follow-on study using broader, more representative populations would sharpen or challenge every conclusion here. The open question I'd most want answered: does ChatGPT's forthcoming persistent memory feature change the usage patterns the researchers observed — and does it change safety failure rates?

• Wikipedia: PHQ-9 depression screening tool (background on the questionnaire used in the ChatGPT study)
• Wikipedia: Reinforcement learning (background on the AI approach used in the psychiatric disorder study)
• Wikipedia: DSM-5 (background on the diagnostic manual used in the safety audit)