News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups
Hosted by Robert Michel

News, Analysis, Trends, Management Innovations for
Clinical Laboratories and Pathology Groups
Hosted by Robert Michel
Sign In

University of Florida Study Determines That ChatGPT Made Errors in Advice about Urology Cases

Dec 15, 2023 | Laboratory Instruments & Laboratory Equipment, Laboratory News, Laboratory Resources, Precision Medicine

Research results call into question the safety and dependability of using artificial intelligence in medical diagnosis, a development that should be watched by clinical laboratory scientists

ChatGPT, an artificial intelligence (AI) chatbot that returns answers to written prompts, has been tested and found wanting by researchers at the University of Florida College of Medicine (UF Health) who looked into how well it could answer typical patient questions on urology. Not good enough according to the researchers who conducted the study.

AI is quickly becoming a powerful new tool in diagnosis and medical research. Some digital pathologists and radiologists use it for data analysis and to speed up diagnostic modality readings. It’s even been said that AI will improve how physicians treat disease. But with all new discoveries there comes controversy, and that’s certainly the case with AI in healthcare.

Many voices in opposition to AI’s use in clinical medicine claim the technology is too new and cannot be trusted with patients’ health. Now, UF Health’s study seems to have confirmed that belief—at least with ChatGPT.

The study revealed that answers ChatGPT provided “fell short of the standard expected of physicians,” according to a UF Health new release, which called ChatGPT’s answers “flawed.”

The questions posed were considered to be common medical questions that patients would ask during a visit to a urologist.

The researchers believes their study is the first of its kind to focus on AI and the urology specialty and which “highlights the risk of asking AI engines for medical information even as they grow in accuracy and conversational ability,” UF Health noted in the news release.

The researchers published their findings in the journal Urology titled, “Caution! AI Bot Has Entered the Patient Chat: ChatGPT Has Limitations in Providing Accurate Urologic Healthcare Advice.”

Russell S. Terry, MD

“I am not discouraging people from using chatbots,” said Russell S. Terry, MD (above), an assistant professor in the UF College of Medicine’s department of urology and the study’s senior author, in a UF Health news release. “But don’t treat what you see as the final answer. Chatbots are not a substitute for a doctor.” Pathologists and clinical laboratory managers will want to monitor how developers improve the performance of chatbots and other applications using artificial intelligence. (Photo copyright: University of Florida.)

UF Health ChatGPT Study Details

UF Health’s study featured 13 of the most queried topics from patients to their urologists during office visits. The researchers asked ChatGPT each question three times “since ChatGPT can formulate different answers to identical queries,” they noted in the news release.

The urological conditions the questions covered included:

The researchers then “evaluated the answers based on guidelines produced by the three leading professional groups for urologists in the United States, Canada, and Europe, including the American Urological Association (URA). Five UF Health urologists independently assessed the appropriateness of the chatbot’s answers using standardized methods,” UF Health noted.

Notable was that many of the results were inaccurate. According to UF Health, only 60% of responses were deemed appropriate from the 39 evaluated responses. Outside of those results, the researchers noted in their Urology paper, “[ChatGPT] misinterprets clinical care guidelines, dismisses important contextual information, conceals its sources, and provides inappropriate references.”

When asked, for the most part ChatGPT was not able to accurately provide the sources it referenced for its answers. Apparently, the chatbot was not programmed to provide such sources, the UF Health news release stated.

“It provided sources that were either completely made up or completely irrelevant,” Terry noted in the new release. “Transparency is important so patients can assess what they’re being told.”

Further, “Only 7 (54%) of 13 topics and 21 (54%) of 39 responses met the BD [Brief DISCERN] cut-off score of ≥16 to denote good-quality content,” the researchers wrote in their paper. BD is a validated healthcare information assessment questionnaire that “provides users with a valid and reliable way of assessing the quality of written information on treatment choices for a health problem,” according to the DISCERN website.

ChatGPT often “omitted key details or incorrectly processed their meaning, as it did by not recognizing the importance of pain from scar tissue in Peyronie’s disease. As a result … the AI provided an improper treatment recommendation,” the UF Health study paper noted.

Is Using ChatGPT for Medical Advice Dangerous to Patients?

Terry noted that the chatbot performed better in some areas over others, such as infertility, overactive bladder, and hypogonadism. However, frequently recurring UTIs in women was one topic of questions for which ChatGPT consistently gave incorrect results.

“One of the more dangerous characteristics of chatbots is that they can answer a patient’s inquiry with all the confidence of a veteran physician, even when completely wrong,” UF Health reported.

“In only one of the evaluated responses did the AI note it ‘cannot give medical advice’ … The chatbot recommended consulting with a doctor or medical adviser in only 62% of its responses,” UF Health noted.

For their part, ChatGPT’s developers “tell users the chatbot can provide bad information and warn users after logging in that ChatGPT ‘is not intended to give advice,’” UF Health added.

Future of Chatbots in Healthcare

In UF Health’s Urology paper, the researchers state, “Chatbot models hold great promise, but users should be cautious when interpreting healthcare-related advice from existing AI models. Additional training and modifications are needed before these AI models will be ready for reliable use by patients and providers.”

UF Health conducted its study in February 2023. Thus, the news release points out, results could be different now due to ChatGPT updates. Nevertheless, Terry urges users to get second opinions from their doctors.

“It’s always a good thing when patients take ownership of their healthcare and do research to get information on their own,” he said in the news release. “But just as when you use Google, don’t accept anything at face value without checking with your healthcare provider.”

That’s always good advice. Still, UF Health notes that “While this and other chatbots warn users that the programs are a work in progress, physicians believe some people will undoubtedly still rely on them.” Time will tell whether trusting AI for medical advice turns out well for those patients.

The study reported above is a useful warning to clinical laboratory managers and pathologists that current technologies used in ChatGPT, and similar AI-powered solutions, have not yet achieved the accuracy and reliability of trained medical diagnosticians when answering common questions about different health conditions asked by patients.

—Kristin Althea O’Connor

Related Information:

UF College of Medicine Research Shows AI Chatbot Flawed when Giving Urology Advice

Caution! AI Bot Has Entered the Patient Chat: ChatGPT Has Limitations in Providing Accurate Urologic Healthcare Advice

In 2017, to Offset Declining Reimbursement and Shrinking Budgets, Savvy Clinical Laboratories Are Using LEAN to Improve Service and Intelligently Cut Costs

Sep 21, 2016 | Compliance, Legal, and Malpractice, Laboratory Instruments & Laboratory Equipment, Laboratory Management and Operations, Laboratory News, Laboratory Operations, Laboratory Pathology, Laboratory Testing, Management & Operations

Nation’s most experienced lab operations managers, cost-cutters, and Lean experts will gather to share successes and proven ideas at Lab Quality Confab on October 18-19, 2016

Most hospitals and health systems are in the first stages of developing their budgets for 2017. Clinical laboratory administrators and pathologists at these institutions report three common factors are driving the next budget cycle: falling reimbursement, flat or declining inpatient admissions, and directives to cut their lab budgets.

“At our health system, the challenge is a bit different,” said one lab administrator at a large Midwest hospital. “Inpatient volumes are increasing, but we get less money from health insurers per admission. For that reason, our budget planning requirement is to accept a smaller budget than last year, while planning to handle more specimen volume in 2017, compared to this year.” (more…)

Energetic Microbiologist-Turned-Ambassador Puts Out a Call to Action for Medical Laboratory Volunteers for Haiti, Guatemala, and the Dominican Republic

Feb 22, 2012 | Instruments & Equipment, Laboratory Operations, Laboratory Pathology, Management & Operations

From volunteer services, to replaced equipment, to outdated NCCLS materials, anything can be of help in poor countries where no medical laboratories come anywhere close to those of the caliber many of us take for granted.

Carla Orner never sleeps. No one as busy as she is has time to waste on even a little shut-eye. She is the full-time ambassador for Heart to Heart International. Her relationship with Heart to Heart International (HHI) began during her attendance at a regional meeting of a medical laboratory organization. “A speaker who was a HHI employee asked for medical laboratory volunteers to assist in its mission,” she says. The rest is history, as the saying goes!  She works with doctors and nurses who volunteer, but her primary goal is to attract more medical laboratory technicians and technologists to join the volunteer effort through Heart to Heart. One tip that Orner shares with potential volunteers is that of the “mobile” CLIA license, which allows the establishment of a lab that can be operated anywhere in the United States.  In all her experience in filling out forms for CLIA, Orner confessed, “I never saw the box labeled ‘mobile.’”

Orner also continues to present at CLMA and ASCP, among other organizations’ annual and regional meetings.  For many years, she held a position as general manager of Regional Laboratory Alliance in Kansas City, MO, where she led an integrated network of community based hospitals and independent reference laboratories. Her 36 years of laboratory experience included night shift, evening shift, and 15 years microbiology. Among all of that, Orner was awarded a B.S. in Medical Technology from Central Missouri State University, and an MBA from MidAmerica Nazarene University. (more…)

UK’s Association for Clinical Biochemistry Calls for Better Blood-draw Training for ED Doctors

Dec 12, 2011 | Compliance, Legal, and Malpractice, Laboratory News, Laboratory Operations, Laboratory Pathology

Studies show clinical laboratories still grapple with sub-optimal specimens from emergency departments and better phlebotomy skills are part of the solution

Improving the quality of medical laboratory specimens collected by the staff of emergency departments is an ongoing goal at most American hospitals. Now everyone associated with phlebotomy will be interested in a study released in the United Kingdom (UK) that recommends that emergency department doctors in that country would benefit from a refresher course on correct specimen collection technique.

Clinical laboratory managers and phlebotomists in most developed nations are well acquainted with the problem of faulty specimens sent from the emergency department. That is the problem highlighted by this UK study. (more…)

Clinical Laboratory Leader from Uganda Wins Scholarship, Takes New Knowledge Back to Uganda

Jul 11, 2011 | Laboratory News, Laboratory Pathology

Scholarship program for aspiring clinical laboratory managers helps them sharpen their skills

Over in Africa, one of Uganda’s main clinical laboratory organizations is about to go “Lean.” Credit for that development goes to one intrepid medical laboratory leader and his trip across the Atlantic to participate at the Executive War College on Lab and Pathology (EWC) that took place in New Orleans last May.

Faithful readers of Dark Daily will remember Ali Elbireer, MT (ASC). He was this year’s winner of a unique clinical laboratory education scholarship that is awarded annually by The Dark Report and Medical Laboratory Observer. This scholarship is designed to advance the medical laboratory management skills and careers of the clinical laboratory industry’s most promising “up and comers.” (See Dark Daily, “ Teaching the Next Generation of Clinical Pathology Laboratory Managers, April 11, 2011“.)

(more…)

;