By Dr Jeremy Gauntlett-Gilbert; Consultant Psychologist and student, MSt Practical Ethics
Chatbots are increasingly being used to deliver an AI version of psychological therapy. Internationally, there is pressure on mental health services and a shortage of human therapists. Mental health providers have a keen interest in such AI technological solutions that might offer treatment “at volume” at low cost. Several tech companies have received, formal ‘healthcare device’ approval for their AI therapy chatbot models – for example, Wysa in the USA, and Limbic in the UK.
Therapy chatbots are often perceived, by users, as highly empathic, and as genuine partners in treatment. This is achieved by chatbot responses like the following, taken from the Limbic website: “I understand, James. Anxiety can be incredibly challenging”; “I’m here for you” (this system is deployed by the NHS in the UK).
These responses can clearly be comforting. But what is their truth value? No-one really thinks that chatbots understand the experience of anxiety. After all, they are text-generation systems that do not even know that they are ‘talking’ to a person, let alone an individual called James. Although well-intentioned, statements like “I understand” cannot be true. Similarly, text such as “I’m here for you” is comforting, but suspect. A more truthful response might be: “this model will reliably generate empathic-sounding text whenever you prompt it, day or night”. The implication of human-like care or support is not true. The chatbot is simply designed to say what a caring human would say, in a personalised way, at relevant moments. If our alarm clock wakes us regularly in the morning, we don’t ascribe ‘helpfulness’ to it or thank it for being ‘supportive’. It is simply doing what it has been made to do. We should have the same clarity about chatbots, although the clever, personalised text that they produce encourages us to imagine more.
So, it seems there is some kind of truth deficit, in the text that is given to users. Where exactly is the truth problem? Clearly, chatbots aren’t capable of ‘lying’ or ‘misleading’ anyone, as they are computer systems, not moral agents. To keep things simple, I will just look at the actual text produced by therapy chatbots. I argue that there is an existing philosophical category that fits it well. The most useful concept seems to be that of ‘bullshit’, created by Harry Frankfurt, and widely used in the philosophy literature.
Bullshit is a type of untruth that is different from lying; liars still care about the truth and are intending to falsify it. Instead, bullshit has been defined as a mixture of truth and lies that is primarily designed to create an effect in the listener (most obvious in the political sphere). It is, in fact, is indifferent to the truth value of what has been said, in the pursuit of the goal. Bullshitters usually imply that they have a much higher level of knowledge than they actually do.
We can examine the mixture of truths and untruths in chatbot statements. Let’s analyse “I understand, James. Anxiety can be incredibly challenging.” For a start, the use of “I” is untrue – it implies that the model is a person or subject, when it isn’t. The use of the first-person pronoun is well known as a developer’s tool to produce anthropomorphic impressions in users. As we have noted, “understand” is untrue, as is “James”. The chatbot is not ‘addressing the user by name’. Instead, it simply knows that “James” is the best word to place in that sequence of words. And then – “Anxiety can be incredibly challenging” – is true, and indeed an important truth for mental health service users and clinicians.
So, there is an admixture of truths and untruths, bundled together in rapid adjacent sequence. They are all designed to create an impression of empathy and support – to ‘create an effect’, in the definition above. The model creates the impression of a level of knowledge of human affairs (i.e. understanding anxiety) that untrue because it is simply a piece of technology. But the text generated is not primarily about representing the truth.
Thus, in technical terms, I argue that these chatbots are producing bullshit. It may be experienced as warm and supportive, but in truth-terms, it remains bullshit. In the broader literature on chatbots, this has been noted in other contexts, with papers choosing such blunt titles as “ChatGPT is bullshit” and “Machine Bullshit”. However, ChatGPT never claimed to be designed for vulnerable users, and nor is OpenAI a trusted healthcare provider. In contrast, Limbic claims this and is deployed by the NHS. Does this context mean that bullshit matters more?
It is almost embarrassing to have to ask the question – “what is wrong with lying to vulnerable patients, if it is helpful?”. However, a great deal of time, energy and legislation have gone into confirming that truth matters. For example, the Psychology regulator in the UK, the HCPC, requires Psychologists to “work within the limits of your knowledge and skills” and to “be honest and trustworthy”. Failing to do this result in a professional being ‘struck off’ and losing their ability to practice. There is no clause that says, “but it is ok to bend these rules and to bullshit a bit, if it seems to help, and the patient likes it”.
There are clear instrumental harms of untruth: treatment is unlikely to work, long term, when not based in truth, the untruth may be ‘found out’ with negative consequences for the patient, such as relationship damage and a sense of betrayal. Also, untruth corrodes trust in entire professions, undermining effectiveness. However, the normative issue is distinct and clear, even if no ‘harm is done’. Generally, it is not OK to lie to people. This surely becomes clearer if the lie is (1) from a trusted professional, (2) during a treatment that is intimate and sensitive and (3) where the patient is known to be struggling and more prone to harm.
It is also ethically and legally clear that professionals and organisations are liable for the truthfulness and actions of their chatbots. In a well-known example, an Air Canada chatbot gave a customer the wrong advice, and they lost out; when the customer sued, the company tried evade responsibility by blaming the chatbot. This defence was categorically rejected; the company was held responsible for its tech. Similarly healthcare organisations that provide chatbots as a service, and professionals who ‘prescribe’ them, retain responsibility for recommending bullshit-producing devices, any harm that comes from this, and for the promotion of untruth in their services.
Practically, this means that we cannot solely evaluate this technology by asking – does it seem to work for some people, and do they seem to like it? These may be necessary criteria for chatbot deployment, but they are not sufficient. We need ongoing normative analysis of why professionals are held to such high truth-standards, and hard questioning about whether it is ok to suspend these when it comes to chatbots.