Artificial intelligence (AI) is changing at an incredible pace. Its rapid, unregulated integration into healthcare raises legitimate concerns for patient safety and the overall efficacy of these novel technologies. Dr. Yves Saint James Aquino, a research fellow at the Australian Centre for Health Engagement, Evidence and Values at the University of Wollongong, emphasizes that patients must be informed whenever digital scribe technology is utilized, even if they have previously consented. This recent guidance underscores the urgent need for transparency in the integration of AI tools into clinical practice.
According to the Australian Medical Association (AMA), nearly one in four general practitioners (GPs) across the country are already using AI digital scribes. They use these tools just for note taking. These tools utilized generative AI, specifically large language models (LLMs) to provide summaries of large texts, though questions about their regulatory status and accuracy remain. Further, Dr. Aquino highlights that most software applications do not meet the legal definition of a medical device. This definition is governed by the Therapeutic Goods Administration (TGA).
The Need for Informed Consent
Dr. Aquino firmly agrees that informed consent is very important for patients using digital scribe technology. He states, “Patients have to be informed every time it [digital scribe technology] is being used,” which underscores the ethical responsibility of healthcare providers to keep patients aware of how their information is being processed.
This consent conversation comes at an important time, as the harms and complexities of AI technologies remain contentious. Dr. Aquino feels people deserve to be able to look at the results from these systems. In this manner, they’ll be able to check the correctness of their medical history. He notes, “So, what consumers and patients should know is that a lot of AI scribes are not really tested in the same way that we expect a medical product, a pharmaceutical product or a medical device should be tested.”
The call for transparency does not stop at informed consent. It needs us to have a strong grasp of the data that fuels these AI systems. As Dr. Aquino wrote recently, large language models poorly depict racially and ethnically diverse people. This creates some alarming questions about the lack of inclusivity and the potential for biased results.
Regulatory Challenges
The TGA’s most recent report was released just last month. It further illustrates the dangers that result from the rapid fire deployment of artificial intelligence innovation into clinical practice. In the Agency’s view, as many AI software applications do not meet the definition of a medical device. This leaves a significant regulatory gap, which has the potential to jeopardize the safety of patients.
To Dr. Aquino that means, “robust testing and regulatory scrutiny.” Without them, we’ll never know the long-term impacts of introducing these technologies into healthcare. He warns, “It means we don’t really know the implications or we don’t really know any evidence of safety or long-term harm.”
This absence of regulation raises concerns for the health care community and patients as well. Dr. Aquino has touched upon something significant. Specifically, developers and users of the app need confidence that any data stored or processed outside of Australia has equivalent protections and privacy laws.
Quality of Training Data
Patient safety advocate Jennifer Morris has been an outspoken activist on these matters. She’s the author of a recently released playbook on AI technology intended for consumer and community advocates. Morris reiterates that the best AI systems in the world are worthless if not trained using high-quality data.
She even fact-checked some of the AI-generated letters and voiced disapproval over their lack of accuracy. “I decided it’s not meeting my standard of accuracy,” she remarked, highlighting the potential risks associated with relying on inadequately tested AI systems.
Morris further states that patients possess the right to access their medical records and evaluate their accuracy. “They have the right to access their medical records and decide for themselves whether they think it’s accurate enough.” This statement further underscores the need for patient empowerment in this new era filled with digital health technologies.
