Photo courtesy of University of New South Wales
Amid rising concerns over AI companions' links to harmful advice, dependency, and suicide, researchers at the University of New South Wales have developed prototype digital companions for students experiencing loneliness, with safeguards aimed at reducing unsafe interactions and overreliance.
Just last month, a United States lawsuit was filed against tech giant Google over a chatbot interaction tied to a teenager's death by suicide. The case, along with other recent reports of emotional dependency and harmful exchanges involving AI companions, has heightened scrutiny of the technology's risks.
At UNSW Sydney, a multidisciplinary research team has introduced two screen-based AI companion prototypes, Tom and Mia, to help students process and regulate difficult feelings.
They were developed with input from Chinese students at the university, drawing on what the research team describes as "lived experience" data.
According to UNSW Sydney, the companions, which can speak both English and Mandarin, were developed to address loneliness, which is increasingly recognised as a public health issue affecting physical and mental health, quality of life, and life expectancy. They are intended to provide skilled, friend-like conversation rather than replace therapists or human companionship.
Tom and Mia are part of a suite of AI companions developed at the felt Experience & Empathy Lab for users such as people in aged care and those living with dementia and other neurodegenerative conditions. The prototypes remain under iterative testing and have not yet been formally trialled.
In an interview with Mobihealth News, UNSW Scientia Professor Jill Bennett discussed how her team is designing the AI companions with safeguards against harmful behaviours seen in other platforms. She also outlined the project's co-design approach, early testing insights, non-clinical limits, and the challenges around measuring mental health outcomes and cross-cultural use.
Q. How do you qualify "lived experience?" How does the data differ from that publicly collected by popular consumer AI chatbots/LLMs? Compared to the breadth of data those popular chatbots learn from, how do your AI companions ensure they can offer support to lonely users?
A. Good question – we are a psychosocial design/research team with a focus on lived experience, which means we use bottom-up qualitative methods to find out about the texture and context of experience. We don't presume anything, but work intensively with a given community or user group to find out what they feel, what is missing, and what they want, as well as understand how technology fits into their lifestyle. But within a psychosocial design framework, we don't simply focus on giving people what they say they want in the short term but on larger goals. For example, if someone said they wanted to feel numb or be transported to a fantasy world, that wouldn't be the goal of AI companion use, which might focus on uncovering the reasons for those feelings and a wider range of options.
Q. Recent reports have linked AI companions to cases of emotional dependency, harmful advice, and even suicide. What specific guardrails or intervention protocols have you built into Tom and Mia to prevent escalation in high-risk conversations, especially when a user expresses self-harm ideation, or users develop unhealthy emotional reliance or substitute AI interaction for real-world relationships? Are there escalation pathways to human support?
A. Many of these problems arise from the in-built goal of LLMs, which is to prolong conversation as long as possible. Excessive validation and agreeability are a means of doing that, although ChatGPT pulled back on this somewhat following some tragic outcomes and legal cases.
By contrast, we focus on building the AI's capacity to gently challenge rather than validate negative beliefs.
Tom and Mia are prototypes currently being iteratively developed and tested by students, so they are not yet a finished product, but we would agree on specific escalation protocols, balancing safety and confidentiality with clients.
Another thing to bear in mind is that it can be unsafe to shut down a conversation abruptly, so as far as possible, we train AI companions in techniques for staying with a user while obviously identifying appropriate external supports.
Q. Where do you draw the line on use cases? For instance, are there explicit restrictions on discussing topics like self-harm, substance use, or trauma, and how does the system handle users who persist in those conversations?
A. As above. Explicit restriction may not be helpful if the goal is to address loneliness and people aren't getting the help they need. Often, people are quite upfront or provocative with chatbots when they would be far more reserved with humans – we can see this as an opportunity to ensure people receive the support they need. If the AI companion can perform as a confidante with a level of skill, encouraging people to seek professional help when needed, this is valuable.
Q. Beyond design intent and co-creation, what empirical evidence do you have so far that these companions actually reduce loneliness or improve mental health outcomes, and not just increase engagement or usage time?
A. We have recently run a successful formal evaluation in aged care for similar AI companions in a very different context, but we haven't formally trialled Tom and Mia. This is actually a point of difference for us: we engage in extensive codesign research and iterative testing before launching a product. That is the only way to design AI to reduce loneliness and improve wellbeing – and as you suggest, to understand and avoid overreliance. You can evaluate a given product in terms of impact, but that won't tell about the mechanisms and processes leading to an outcome/effect or how to design a better outcome.
These companions are not delivering clinical therapy, although we may borrow therapeutic techniques to model good responses in their training. They are more like peers with acquired skills, so they can talk about everyday situations – the movie you saw, the social event you went to, whatever anxieties arose from it – with a level of competence. But ideally, the goal of this is to engender some perspective and insight in the users by facilitating reflection. This is something we all benefit from when we have worries.
You also mention trauma. Many people have histories of trauma, loss, and bereavement that are impactful but may not require clinical attention. Grief is something that often comes up. It may be something that is well managed on the outside and that people don't want to share in their social or work environments, but there can be an ongoing emotional toll that needs to be spoken about or processed. There are many instances where people have used AI companions very effectively for this purpose.
Q. At this research stage, what failure modes have you already observed, for example, inappropriate agreement or mishandling of distress? How are you testing and mitigating these risks before any real-world deployment?
A. We have worked to mitigate excessive agreement or validation as mentioned. Generally, the AI companions are quite good; if anything, the problem can be offering too much good advice too soon because they lack the human sense of judgment. It's one thing to know what is good for someone, another to persuade them without overwhelming or bombarding. So, often it's a question of training the companion to slow down a bit. We have done a lot of work on pacing and attunement, especially with older users.
But I want to add a very important point here. We are seeing legislation emerging – for example, in China – around the regulation of anthropomorphic AI that simulates human companionship. I think it is very important that AI doesn't conceal its AI status, and that people remain cognisant of the difference between AI knowledge and human knowledge. There are concerns about the long-term effects of ultra-realistic products, especially those that are marketed as "romantic" companions. It is not simply that these may promote delusion; they reinforce stereotypical fantasies of ideal girlfriends, for example, that are impossibly perfect and compliant, which does not help users to forge relationships with real humans. AI companions are not a substitute for a human companion, and so, while we want to enable them to safely and effectively support users, the goal may not be to make them indistinguishable from humans.
Q. Since the companions operate in both English and Mandarin, how do you ensure consistency in tone, safety responses, and cultural sensitivity across languages, especially in emotionally charged conversations?
A. This is an interesting question that, in fact, adds depth to the project. One of my PhD students, who is Chinese and has trained in psychoanalysis, talks about how it is difficult for students to find their "emotional self" in a second language. This is a good example of how lived experience is incorporated. If students say, it's hard to "be yourself" in English, this is the kind of thing that can be played out and explored – as opposed to perfecting speech technically, which is relatively easy by comparison.
_
Prof Bennett's responses have been edited for clarity.
