by Keila DePape, McGill University

Credit: Pixabay/CC0 Public Domain

AI-powered apps offering medical diagnoses at the click of a button are often limited by biased data and a lack of regulation, leading to inaccurate and unsafe health advice, a new study found.

McGill University researchers presented symptom data from known medical cases to two popular, representative apps to see how well they diagnosed the conditions. While the apps sometimes gave correct diagnoses, they often failed to detect serious conditions, according to findings published in the Journal of Medical Internet Research. This potentially resulted in delayed treatment.

The researchers identified two main issues with the health apps they studied: biased data and a lack of regulation.

Bias and the ‘black box’ phenomenon

The bias issue is known as the “garbage in, garbage out” problem.

“These apps often learn from skewed datasets that don’t accurately reflect diverse populations,” said Ma’n H. Zawati, lead author and Associate Professor in McGill’s Department of Medicine.

Because the apps rely on data from smartphone users, they tend to exclude lower-income individuals. Race and ethnicity are also underrepresented in the data, said the authors. This creates a cycle where an app’s assessments are based on a narrower group of users, leading to more biased results and potentially inaccurate medical advice.

While apps often include disclaimers stating they do not provide medical advice, the scholar argues that users’ interpretations of these disclaimers—if read—do not always align.

The second issue is the “black box” nature of AI systems, where the technology evolves with minimal human oversight. Zawati said lack of transparency means even an app’s developers may not fully understand how it reaches conclusions.

“Without clear regulations, developers aren’t held accountable, making doctors reluctant to recommend these tools. For users, this means a potential misdiagnosis is just a click away,” said Zawati, who is also an Associate Member in McGill’s Department of Equity, Ethics and Policy and the Faculty of Law and Research Director of the Center of Genomics and Policy in McGill’s Department of Human Genetics.

Call for AI oversight

To overcome limitations, developers can train apps on more diverse data sets, conduct regular audits to catch biases, enhance transparency to improve understanding of how algorithms work and include more human oversight in the decision-making process, he suggested.

“By prioritizing thoughtful design and rigorous oversight, AI-powered health apps have the potential to make health care more accessible to the public and become a valuable tool in clinical settings,” Zawati said.

More information: Ma’n H Zawati et al, Does an App a Day Keep the Doctor Away? AI Symptom Checker Applications, Entrenched Bias, and Professional Responsibility, Journal of Medical Internet Research (2024). DOI: 10.2196/50344

Journal information:Journal of Medical Internet Research

Provided by McGill University

---

Explore further

When AI plays favorites: How algorithmic bias shapes the hiring process

ReplyReply to allForward