Two and a half years ago, a 14-year-old American boy took his life after a last conversation with the “woman” he was in love with, a fictional character created with artificial intelligence (AI), with whom he had a longstanding “relationship” through a popular chatbot. Months later, the boy’s mother filed a lawsuit against the company that had created the platform.
I won’t go into both sides’ legal arguments here. Some reflect the anger and pain of a mother who has lost her child, and the other, the cold response of a company whose business, in part, consists of defending itself against accusations like this. I don’t know what will happen during the trial, which looks like it’s scheduled for November of this year. I imagine that much of civil society, frightened by the extreme scope of AI, will side with the mother and push to prove her right. On the other hand, I think that the company may try to turn the trial into a kind of accusation against the woman and her ex-husband, the boy’s father, for neglecting the boy. We’ll see.
When I heard the story, my first impulse was a furious judgment: that the company is to blame for making available to young people characters with whom they can have “relationships” of all kinds (“friendly,” “loving,” “cybersexual”…) without the proper warnings, age filters, and the necessary reactions of assistance and help. Then I immediately started blaming the parents, saying they should take care of their children, monitor the content they browse, and restrict their screen time.
But it happens that, as I sit down to write, it is natural for me to delve a little beyond the first impression and the fleeting opinion.
So, compassion and a slightly less impulsive reflection led me to consider that the tragedy began long before the moment the boy started using AI. His parents were divorced; the young man had been severely depressed for years, and no one seemed to see or hear it properly. In recent months, when it finally became noticeable, the signs were the usual ones: isolation, abuse of electronic devices, and drastic changes in school behavior. Unfortunately, he was treated with the same usual uninformed strategy: imposing disciplinary measures.
I mention the separation of parents (some will think tendentiously, with a clear “anti-divorce” position) because I consider that, in the life of children and young people, divorce, in fact, always represents a blow that cannot be avoided. Society has supported it as a legal procedure, no doubt because of the amount of abuse that occurs within failed couples, which end up seriously affecting all family members, including, of course, children. However, that does not mean that the pain of parental separation does not result – in practically all cases, with the exceptions that confirm the rule – in a wound that we can describe as infinite for the simple fact of being irreparable. The separation of our parents implies the tearing apart of the symbolic union that gave each of us unity. It can be alleviated but not resolved. In certain cases, it may be better to divorce, but divorce is never good, at least not for the children; certainly not in our society.
Undoubtedly, blended families offer the possibility of recovery when they are established as an environment that becomes stable and healthy for the little ones. In our young boy’s case, at least his mother had remarried and already had two other children. But the new arrangement did not seem to have suited the boy: he spent half the time with his father and half with his mother, which is – in my opinion – a model that only works when the separation has been processed in a dignified way and both environments are stable shelters, where the children are fully received on both sides and feel that the parents want to be with them equally for the same amount of time. But this does not generally happen in a divorce; instead, children perceive that they do not fit in one family or the other and that, if their parents take turns, it is so they can enjoy “childless” time for their personal occupations. The feeling of rejection and abandonment can be intensified for an only child – like our young man – and, even more so, when there are half-siblings from a second marriage who – in painful contrast – do enjoy home as stable.
The young man spent his last weeks chatting with his artificial girlfriend (who, in the fiction created by him and allowed by “her,” was also his twin sister) about how bad he felt, how lonely he was, and that he desired to commit suicide. Regarding the latter, when he said it outright, the machine disapproved with an emotional and empathetic tone; not so when the boy alluded to suicide through metaphors. Then, the chatbot did not understand the double meaning and played along, like in an innocent romance. In their last dialogue, the metaphor he used to talk about dying was that of returning home, that is, to the place where she, his lover, awaited him. The chatbot replied without grasping what he meant, “Do it, please, my handsome king.”
And he did.
What real responsibility does a machine have if someone commits suicide? Or, looking at it another way, what are the real chances it can avoid it? I’m seriously wondering. Telephone assistance services provided by real people can be effective. The key, obviously, is the commitment of the person on the other end of the line to understanding the caller’s suffering. The possibility of success depends, in the first instance, on being authentically sensitive to two facts: that human beings need to establish relationships with real people and that what characterizes such a relationship is that the other person can give something of themselves for us. Giving, risking, or even sacrificing are ways to make ourselves present when someone needs a genuine encounter.
Can a machine do something like that? It seems not, so far. When all appeals to real people have failed, nothing can be done by a machine, behind which there is no human being capable, as I say, of giving something for us. The telephone and Zoom, although they are virtual resources, do allow this type of meeting. But AI chats have an initial condition that, if you think about it, challenges any authentic human relationship: you must start by turning them on.
No one can feel close to someone whom he can turn on and off at will. How can we expect such an entity to give us something of themselves, someone whom we have there, at our beck and call, all the time, someone who never exercises any initiative for our encounter? This is the big difference between chatbots and real human conversation, even those that take place remotely and over periods of time, such as on WhatsApp. Not even young people, with their much-mentioned lack of brain maturity, can fall into such confusion. Only someone with a deep narcissistic delusion, gestated in early childhood, can want to enter a romantic relationship with a being at their beck and call.
This doesn’t seem to be the case with our boy. His behavior was not that of someone suffering from psychotic delusions. It would seem rather that – aware and enraged by his loneliness and abandonment – he played a perverse game of self-harm with the machine, perhaps yes, in the unconscious hope that someone would read his chats or that the chatbot would suddenly provide an unexpected solution.
Unfortunately, his parents and stepfather did not hear the call clearly. For its part, as is obvious, the chatbot only had the resources to react awkwardly and play along.
It is precisely at this point that, as we understand it, the debate on the responsibility of companies that create this type of machine can arise, because they make conversational chatbots with human language available to the public to respond, without conditions, to all the users’ emotional and mental states, whether or not the machines are aware that the conversation is fiction.
We all agree on the urgency of setting mental health priorities for artificial intelligence and implementing all the cyber resources necessary to address them. Among these priorities are, of course, alerts to users in case of suicidal thoughts and, before that, fine-tuning the language detectors of conversational robots so they capture the subtleties with which human beings express their emotional states. A chatbot that establishes intimate relationships must be able to understand the subtlety of a young person who, saying “I want to go home,” implies “I want to die.”
The issue is as urgent as it is complex because it involves providing the machine not only with a language for detecting infinite metaphors, but also with response resources that can guide a disoriented user, and, in addition, doing it subtly, that is, preserving an empathetic communication style and not evidencing the alert with a reaction such as “If you are having suicidal thoughts, please contact these phone numbers….”
However, the very fact that they are machines leads us to think that whatever their response to the user’s emotional need, they will fall short. I know of only one case of someone who activated a machine and, as a result, found solace in the face of an imminent suicidal impulse. I am referring to the story of a dear friend of a loved one of mine, who confessed it to him. The woman was about to take her life when, in search of one last companion, she put on one of Beethoven’s piano sonatas on her sound system. Listening, she cried from a deep place and changed her mind. The late Beethoven, his music, and, of course, the possibility of listening to it at home on an electronic device saved her life.
However, this occurred in the realm of art and beauty, we might believe. I would dare to say that this is the only place where humans can find another human being in an inert object, in this case, a composer who died many years ago. (There is also the world of religion and ritual, but these utilize sacred objects, and I am not sure that we can call them inert). The sensation of encounter that a work of art allows is clear: the creator has given something of themselves. On many occasions, they have even stripped themselves of something to give to us, a surrender with overtones of risk and even sacrifice. (Such works come to acquire a sacred character, such as, I believe, the series of “black paintings,” created by the American painter Mark Rothko at the end of his life.)
Well, something similar will have to happen with artificial intelligence if we really want to create devices that help people overcome the crises of seemingly infinite pain. AI will have to reveal the human behind it, not only as a simulation of real language but as a new type of aesthetic language, one that only artificial intelligence can provide us. Regarding AI, if there is one thing we must demand from companies and ourselves, as a society, it is that its design and development contain authentic art and beauty, including beautiful thoughts and beautiful science. Undoubtedly, an important advance has been to hire philosophers and artists to carry out the task, as some companies are already doing. Another important step is to educate our children and students to interact with AI in a way that, in turn, educates it to become a work of art itself. What kind of work will it be? We do not know yet, but we are beginning to understand that the challenge is enormous, since it is a question of sublimating something as dangerous as the simulation of the human. (Regarding this, it is worthwhile, beyond our personal affiliations, to read the encyclical Magnificent Humanity by Pope Leo XIV, one of the most complete and, surely, controversial documents on this topic).
I end by quoting an extremely illuminating paragraph on the enigma of AI’s future. I extract it from Jesse Barron’s report for the New York Times, about our suicidal youngster:
“As these cases begin to flood the U.S. courts, the legal system will have to determine whether AI belongs to the same categories as older technologies—through which humans have always spoken—or whether there is some quality of what AI produces, that which feels as if someone were expressing themselves, legitimately belongs to its own category, a category that right now is elusive because it is yet to be defined.”
Translation by: Daniel Wetta
