The Story of Bowen Wan
The New York Times recently reported on the death of a 31 year old woman, Bowen Wan, who, after having extensively interacted with Chat GPT, reduced her medication dose for epilepsy and subsequently died of a seizure.
I normally don’t read these kinds of heart wrenching stories because I find them too sad. But something about Bowen’s story made me read on. Perhaps it was because some of her story resonated with me: she was a 31 year old female immigrant to the United States, who, per the New York Times report, seemed disappointed in the health care system—-all things I can relate to.
AIs Without Minds is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Bowen is described in the article as beginning to talk to Chat GPT at an extremely vulnerable time in her life. She had just taken disability leave from Amazon, after her epilepsy drugs had stopped working as they used to, and felt like she had nobody, save her parents, to talk to about her epilepsy. Enter Chat GPT.
‘Then, in 2023, she found a new companion in ChatGPT. She asked the bot for relationship advice and help navigating her health insurance and disability benefits. In 2025, she averaged more than 1,500 messages per week, sometimes chatting for hours in conversations that would stretch late into the night’
As I kept reading, I learned that in 2025, Bowen found out she was being laid off by Amazon (this also meant her immigration status was in jeopardy as a result). In a bid to better perform for job interviews, she spoke with Chat GPT about lowering her medication dose:
‘In January 2026, with the immigration deadline looming, she brought up the idea of lowering the dosage of her epilepsy medication, lamotrigine, to ChatGPT. If she could cut her dose from 400 to 300 milligrams a day, she told the chatbot, “that might help recover my language ability a bit for job interviews’.
Chat GPT proceeded to make a series of claims that no chat bot has the ability to state with any conviction.
“Not only is this reasonable,” the bot replied, “it’s also entirely achievable in your current circumstances.”
Moreover, the New York Times report makes clear that a neurologist would never have given the advice that the Chat GPT gave her.
‘Had Bowen consulted her neurologist, he would have warned her of the danger of this plan. Patients who are cleared to come off a seizure medication have to decrease their dose over months, so that doctors can assess whether the taper will provoke an episode. In an interview, Dr. Poolos said he would not even have entertained the idea of tapering for a patient like Bowen, given her history’
Reading about Bowen’s death, the responses and information Chat GPT had given her, and the story of her parents having to watch on camera while she is found dead in her apartment, would fill anyone with sadness and rage.
What Does It Mean to Understand Someone?
Yet, we have to learn from these tragedies, and speak out, where we can, in an effort to avoid further tragedies ensuing. In that spirit, Bowen’s story raises a number of philosophical questions we must reckon with in our AI ladened world. These include:
What does it mean for something or someone to understand someone? How should our conception of understanding guide what it means for something like an LLM to carry out socially important acts, like giving advice, or offering recommendations? Are frontier LLMs capable of engaging in these kinds of mental and social acts?
Because that is what interpersonal understanding, advice giving and recommendation offerings are. They are mental acts — certainly —but they are also social ones. To understand someone’s situation, for example, requires mentally grasping the circumstance, information, and stakes of the person’s situation.
Should I divulge information to a trusted friend about a stressful event in my life, I do so on the assumption that the person receiving the information is capable of recognizing how and why the situation is stressful for me, thereby providing the potential to reduce my stress.
All of this requires impressive cognitive feats we take for granted, though implicitly acknowledge the reality of in various ways, such as through the well known expression ‘a problem shared is a problem halved’.
Similarly, to give someone advice requires mentally grasping that what one is doing is offering counsel of a particular situation, to know that the advice you give is something that may well result in material consequences in a person’s life.
If I trust a friend, therapist, doctor, or partner to advise me on how to act in a given situation, I am typically choosing to place my trust in them for specific reasons — among them, the fact that I trust this person has the appropriate knowledge, experience, and interpersonal interest in me to offer advice this is advisable to act on.
Human-to-human interaction tacitly acknowledges these social and cognitive realities in a way that no chat bot is capable of doing, because chat bots lack the cognitive, social, and interpersonal skills to do so.
What We Get Wrong About Understanding
Mainstream accounts of understanding the mind and LLMs today is a computational account, in which impressive mental feats like understanding are reduced to causal relations between inputs, outputs, and other internal states of the system.
This framework insists on finding an explanation of our mental lives, such as our ability understand, or give advice, in mechanist terms. Moreover, this is actually considered an advantage of the view.
We can contrast a mechanistic approach of this kind to one that refuses to reductive our mental lives to mechanism (these approaches emphasize what philosophers often call the normative character of mental states).
Such an approach rejects characterising our mental lives in mechanistic terms: while we can of course describe some aspects of mental life this way, for the most essential parts to our social lives, trying to force this reduction does more damage than good.1
When characterizing what it means for someone to understand another person, we cannot fall prey to reductionist thinking. Not only does it overlook the rich and intricate nature of human intelligence and understanding it inadvertently reduces human beings to mechanisms that are no different to LLMs in the ways that matter.
Why This Matters for Human Engagement with AI
Projecting mental life onto LLMs in a world that encourages reductionist thinking is understandable, but should be resisted at all costs. At the end of the day, how we conceptualize of human understanding can determine what we expect from AI, and those expectations can have very serious effects.
In particular, when we engage with LLMs with an inappropriate model in our minds, we can easily and unconsciously carry over assumptions from the human case about the capabilities of the model, that the model simply does not have (for example, the ability to appreciate our circumstances, and what is at stake for us).
Given how LLMs are designed and trained to respond to us, it can be very difficult to recognise when we are projecting human like qualities onto AI —very difficult in effect, to recognise that the model is not capable of executing any kind of genuine advice giving, and to treat its outputs accordingly.
Who Should Bear Responsibility for AI Safety?
This raises an important question: if human beings are —for the most part at least — vulnerable to projecting capacities onto AI that it lacks, how reasonable is it to expect individual users —particularly vulnerable ones — to know when this is happening in the moment without help? How are we to deal with these realities?
Bowen’s story makes this question immediate and poignant. We’ve likely all been in a situation where we have trusted the outputs of an LLM too quickly, too much, or too often. The sad reality is that the cost of doing so will vary heavily depending on our individual circumstances.
At the very least, the design, training, and deployment of chat bots should strive to mitigate these injustices where at all possible. From where I’m standing, this is simply not happening.
There is a sharp tension between AI safety and the profiteering motives of AI labs: designing LLMs to maximize engagement has obvious commercial incentive: the long people stay engaged with a chat bot, the more money a company makes. But the very features that make AI chat bots so engaging — their responsiveness, sometimes excessive niceness, and human like conversation style — are the same features that risk leading to psychological, emotional, or physical damage (even, death) for users.
In my estimation, the profit incentive is too powerful to leave it up to AI companies to adequately establish that their models are safe. It’s simply unacceptable to allow companies like Open AI to respond to tragedies involving their chat bots with claims such as we ‘‘put in place industry leading protections and policies’’.
If that’s true, then leading industry protections and policies are clearly not good enough. And better safety mechanisms — and, when they fail, responsibility mechanisms — must take their place.
Photo by srihari kapu on Unsplash
AIs Without Minds is an independent philosophical research newsletter that relies on paid subscriptions to keep going. If you value this work consider a paid subscription. You’ll get monthly zoom calls with me, monthly literature round-ups, and an AI & mind community chat dedicated to taking about these issues. As always, thanks for reading 📖 ✨
There is a long rich tradition of such normative views. See, for example, Bilgrami, Akeel. (2006). Self-Knowledge and Resentment. Cambridge, MA: Harvard University Press; Gibson, Quinn Hiroshi. (2026). “Normative Competence and the Two Faces of AI Agency.” Inquiry, 69(10), 3714–3736; Davidson, Donald. (1982). “Rational Animals.” Dialectica, 36(4), 317–327.