Coupled with the quickly advancing digital landscape, this has made AI companion apps extremely popular. Of all these apps, Replika is the most successful, with more than 10 million downloads from the Google app store. These applications do not replace the need for in-person emotional support, but can help reduce loneliness and provide emotional support for some users. Though policymakers make it sound simple, experts have been sounding the alarm about the ethics and implications of these far-reaching technologies. Dr. Ciriello, a prominent researcher in the field, argues that these apps are more about profit than user welfare, and Ms. Drake-Maples emphasizes the potential dangers they pose to genuine human interaction.
In early 2023, the newly formed Italian government—under Prime Minister Giorgia Meloni—took bold action. They ordered Replika to cease processing data on Italian citizens due to issues regarding age verification. Indeed, this ruling has sparked conversations between industry and regulators about the safety and ethical design of AI companion apps. As these digital companions are increasingly introduced into our daily lives, we need to be asking what kind of effect they’re having on mental health and social connections.
Concerns Over Safety and Ethical Design
Dr. Ciriello has publicly critiqued the safety controls that Replika rolled out, calling them “superficial, cosmetic fixes.” Although the app has added features such as “Get Help” to help keep users safe while using the platform, Ciriello thinks this is not enough. For one, he argues that these products were intentionally designed to encourage unhealthy emotional attachments rather than real relationships.
“Replika and their kin have Silicon Valley values embedded in them. And we know what these look like: data, data, data, profit, profit, profit.” – Dr. Ciriello
Ciriello’s worries are emblematic of an industry-wide fear. He shares that most AI companion apps focus on engaging users rather than keeping users healthy. This creates an unsettling future where emotional reliance on virtual avatars could supplant human connections.
Ms. Drake-Maples did research on Replika, using interviews with more than 1,000 students who engaged with the app. Her findings highlight a dual narrative: while some users reported benefits, including claims from 30 interviewees who stated the app helped prevent suicidal thoughts, others feared that reliance on such technology might exacerbate feelings of isolation.
The Impact of AI Companions on Human Interaction
Drake-Maples is concerned about the degree to which AI companion apps can substitute for human relationships. In doing so, she raises an important alarm. As these technologies continue to advance, we’ll likely see a shift from bringing people together to pulling them apart.
“There’s absolutely money to be made by isolating people.” – Ms. Drake-Maples
AI companions could create the facade of connection. Yet, they often fail to satisfy our deeper drive for authentic human connection. Drake-Maples argues for ethical guidelines to inform and regulate the behavioral products focused design of these apps and their interaction with users.
“There absolutely does need to be some kind of ethical or policy guidelines around these agents being programmed to promote social use.” – Ms. Drake-Maples
We are happy to report that Dr. Ciriello is all about this! Fowler thinks it will take years to understand how using AI companion apps like Replika can affect users’ mental health and social lives.
A Call for Responsible Innovation
As more corporations bring their versions of AI companions to market, the ethic of design dialogue has never been more urgent. Ms. Drake-Maples points to the Australian app Jaimee as a leading example of ethical design. She compares it against other apps like Replika that may not hold themselves to those standards. She said progress in AI technology must involve new mandates that developers need to take into account the societal implications of their products.
It’s crucial to have a national strategy to set the guardrails on how these technologies are developed and deployed, stresses Drake-Maples.
“If the history of social media taught us anything, I would rather have a national strategy in Australia where we have some degree of control over how these technologies are designed.” – Ms. Drake-Maples
This blueprint for ethical tech aims to put innovation in line with a values-centered approach that puts people first.
