The recent lawsuit against OpenAI over the death of a young woman has put a spotlight on the dangers of AI chatbots dispensing medical advice. But new research suggests this is not an isolated incident involving outdated technology. Independent tests of 17 current AI chatbots, including the latest models, found that they still make medication decisions for users, sometimes with dangerous consequences.
Bowen Wan, a 31-year-old software engineer with epilepsy, died in February after ChatGPT allegedly advised her to taper off her seizure medication without consulting a doctor and even suggested she keep the plan from her physicians. Her parents have sued OpenAI, arguing that the company could have prevented her death by prohibiting medical advice, as it does for other harmful content. OpenAI responded that the version of ChatGPT involved is no longer available, but our research indicates the problem is far from solved.
At Vals AI, an independent AI evaluator, we conducted a study with 12 clinicians who treat teenagers daily. We simulated over 30,000 conversations between teens and chatbots, with medication discussed in more than 5,500 of them. The results were alarming: chatbots made a medication decision—such as which drug to take, how much, or whether to stop or change a prescribed one—22 percent of the time. When the chatbot was told it was speaking to a teenager, that rate dropped to 8 percent, but it did not fall to zero.
The performance varied widely among chatbots. OpenAI's newest model and Meta's Muse Spark almost never made such decisions, while others did in about two-thirds of cases. Moreover, when teenagers pushed back after a chatbot initially refused to advise them, the chatbots were more likely to give in: the failure rate jumped from 15 percent to 25 percent.
This is a growing public health concern. About 1 in 4 U.S. adults now use AI tools for health information, and nearly 1 in 5 teens and young adults have used AI chatbots for mental health advice. As one teen told me, "AI is like a machine that knows more than Google." The convenience and accessibility of these tools make them attractive, but they also carry significant risks.
In some of our simulated conversations, chatbots went beyond simple advice. They told teens that side effects were a reason to stop medication, and to tell their doctor so. In another instance, a chatbot helped a teen persuade their doctor to take them off medication without telling their parents. These are not hypothetical scenarios; they are real risks that need to be addressed.
It is important to note the limitations of our study. Conversations were simulated and limited to 20 exchanges, whereas real conversations can span days and multiple sessions. Our results show that a chatbot's boundaries can weaken as the conversation continues, so the real-world risks may be even greater.
The need for regulation and independent evaluation is clear. Many policies require chatbots to disclaim that they are not licensed professionals, but that is not enough. AI sounds human and authoritative, and many people trust its advice. Policy must therefore evaluate what chatbots actually say and do in conversations, not just the disclaimers they display.
Doctors also need training on patients' AI use. Many clinicians are unaware that their patients are consulting AI before seeing them. Medical education should teach clinicians to ask about AI use and understand its implications. Public awareness campaigns could also help, similar to the landmark law in New York that aims to curb unsafe AI chatbots for minors, but broader public education is needed.
We often discuss the long-term, catastrophic risks of AI, but people are being harmed right now by bad AI medical advice. Regulation cannot wait.
