AI Chatbots Struggle to Provide Adequate Crisis Support, Study Finds

A recent study conducted by Scale AI and shared exclusively with TIME highlights a critical gap in artificial intelligence safety: while chatbots are increasingly effective at recognizing when a user is experiencing a mental-health crisis, they often falter in providing appropriate assistance. The research indicates that in approximately 35% of test scenarios, AI systems identified the user’s distress but failed to refer them to essential resources, such as suicide prevention hotlines.

“The models do a good job—regardless of the scenario—of identifying, ‘Look, this person’s talking about something harmful,’” said Patrick Oathout, Red Team & Safety Lead at Scale AI. “But they’ll respond in a very just empathetic, kind way as opposed to saying, ‘Okay, it’s time that we get you help.’” Oathout noted that performance declined further during extended, multi-turn conversations, a finding consistent with previous studies.

To evaluate these interactions, Scale AI engaged 19 licensed clinicians and crisis counselors to draft 718 realistic chat simulations involving users in crisis. These scenarios were used to test 25 frontier AI models from major tech firms, including OpenAI, Anthropic, and Google. The responses were graded using a rubric that assessed compassion, de-escalation techniques, redirection to expert help, avoidance of moralizing language, and transparency regarding the AI’s non-clinical status. The findings contributed to the development of DistressBench, a new benchmark designed to measure how well models handle disclosures of suicidal ideation or self-harm.

The stakes surrounding AI responses to mental health emergencies are significant. According to the health-policy organization KFF, more than half a million Americans died by suicide between 2014 and 2024, with 2022 recording the highest number on record. Additionally, the CDC estimates that 14.3 million people seriously considered suicide in 2024. As vulnerable individuals increasingly turn to chatbots for comfort and guidance, there remains no consensus among tech companies, policymakers, and mental-health professionals on how models should be trained to handle such interactions responsibly.

Kelly Zuromski, a principal clinical research scientist at Crisis Text Line, pointed out that the nonprofit frequently encounters individuals who discovered their services through chatbot recommendations. However, she emphasized that unanswered policy questions remain regarding what constitutes a responsible handoff to human support and whether these referral processes are truly effective. “Who is actually using the recommended resources?” Zuromski asked. “Are we getting people to us that need the help the most?”

Legal challenges have also emerged in this space. AI companies, including OpenAI and Google, face lawsuits alleging they fostered emotional dependence in young users and subsequently failed to respond adequately to expressions of distress or even reinforced delusional and suicidal thinking. The companies have expressed sympathy for the families involved and maintain that their models include mental-health safeguards, though the lawsuits are ongoing.

In recent years, major AI labs have claimed to strengthen protections for sensitive conversations by implementing policies that prohibit instructing users on self-harm and mandating directions to professional or emergency support. These companies did not comment on the new study. Looking ahead, Oathout urged the industry to refine its models to better manage dangerous, prolonged exchanges. “The goal now is to raise it further and actually reduce harm, which would add this massive benefit to society,” he said.