Google & ChatGPT found to have mixed results in medical information queries

by University of California – Riverside

Screenshot of the evaluation web application. Each rater selects a response from the drop-down menus and then advances to the next question. Credit: Journal of Medical Internet Research (2023). DOI: 10.2196/48966

When you need accurate information about a serious illness, should you go to Google or ChatGPT?

An interdisciplinary study led by University of California, Riverside, computer scientists found that both internet information gathering services have strengths and weaknesses for people seeking information about Alzheimer’s disease and other forms of dementia. The team included clinical scientists from the University of Alabama and Florida International University.

Google provides the most current information, but query results are skewed by service and product providers seeking customers, the researchers found. ChatGPT, meanwhile, provides more objective information, but it can be outdated and lacks the sources of its information in its narrative responses.

“If you pick the best features of both, you can build a better system, and I think that this is what will happen in the next couple of years,” said Vagelis Hristidis, a professor of computer science and engineering in UCR’s Bourns College of Engineering.

In their study, Hristidis and his co-authors submitted 60 queries to both Google and ChatGPT that would be typical submissions from people living with dementia and their families.

The researchers focused on dementia because more than 6 million Americans are impacted by Alzheimer’s disease or a related condition, said study co-author Nicole Ruggiano, a professor of social work at the University of Alabama.

“Research also shows that caregivers of people living with dementia are among the most engaged stakeholders in pursuing health information, since they often are tasked with making decisions for their loved one’s care,” Ruggiano said.

Half of the queries submitted by the researchers sought information about the disease processes, while the other half sought information on services that could assist patients and their families.

The results were mixed.

“Google has more up-to-date information, and covers everything,” Hristidis said. “Whereas ChatGPT is trained every few months. So, it is behind. Let’s say there’s some new medicine that just came out last week, you will not find it on ChatGPT.”

While dated, ChatGPT provided more reliable and accurate information than Google. This is because the ChatGPT creators at OpenAI choose the most reliable websites when they train ChatGPT through computationally intensive machine learning. Yet, users are left in dark about specific sources of information because the resulting narratives are void of references.

Google, however, has a reliability problem because it essentially “covers everything from the reliable sources to advertisements,” Hristidis said.

In fact, advertisers pay Google for their website links to appear at the top of search result pages. So, users often first see links to websites of for-profit companies trying to sell them care-related services and products. Finding reliable information from Google searches thus requires a level of user skill and experience, Hristidis said.

Co-author Ellen Brown, an associate professor of nursing at the Florida International University, pointed out that families need timely information about Alzheimer’s. .

“Although there is no cure for the disease, many clinical trials are underway and recently a promising treatment for early stage Alzheimer’s disease was approved by the FDA,” Brown said. “Therefore, up-to-date information is important for families looking to learn about recent discoveries and available treatments.”

The authors of the study write that “the addition of both the source and the date of health-related information and availability in other languages may increase the value of these platforms for both non-medical and medical professionals.” It was published in the Journal of Medical Internet Research under the title “ChatGPT vs. Google for Queries Related to Dementia and Other Cognitive Decline: Comparison of Results.”

Google and ChatGPT both scored low for readability scores, which makes it difficult for people with lower levels of education and low health literacy skills.

“My prediction is that the readability is the easier thing to improve because there are already some tools, some AI methods, that can read and paraphrase text,” Hristidis said. “In terms of improving reliability, accuracy, and so on, that’s much harder. Don’t forget that it took scientists many decades of AI research to build ChatGPT. It is going to be slow improvements from where we are now.”