Researchers at the University of Maryland evaluated how accurately the ChatGPT chatbot conveys information about breast cancer screening. In a study published in Radiology, 88 percent of the responses were found to contain correct and actionable details related to screening guidelines, risks, and practical considerations. The researchers designed a rigorous assessment to test the reliability of AI generated advice in a real medical topic that affects many individuals in Canada and the United States.
To probe the bot’s consistency, the team crafted 25 specific questions covering standard breast cancer screening recommendations. Each question was asked to ChatGPT three times to observe potential variation in answers, a known behavior of the model when it is prompted differently or when its training data updates. The exercise mirrors how patients might interact with AI tools in preliminary health inquiries and raises important questions about consistency in medical guidance.
Three radiologists with mammography expertise reviewed the responses for accuracy, tone, and usefulness. Their assessment found that the answers aligned with evidence for 22 of the 25 questions. The examiners noted that ChatGPT correctly addressed topics on breast cancer symptoms, recognized risk factors, and offered guidance on the frequency of mammograms as well as cost considerations. The study highlighted that, overall, the information was presented in a clear and approachable format, which can support patient understanding when used as a supplement to professional advice.
One response relied on outdated guidance by suggesting a postponement of mammograms for a four to six week period following COVID-19 vaccination. The February 2022 recommendations do not advise delaying the procedure, a lapse that underscores the need for ongoing updating of AI outputs as medical guidelines evolve. A second and a third response demonstrated substantial variation between iterations, signaling caution for relying on AI for precise recommendations in dynamic clinical contexts.
Looking ahead, the research team is extending their evaluation to include lung cancer screening guidelines and the broader landscape of preventive care. They are exploring methods to improve the alignment of AI driven advice with current clinical standards, including more explicit prompting strategies and mechanisms to verify updates against trusted medical sources. The investigators emphasize that the medical community bears responsibility for continually evaluating such technologies and safeguarding patients from potential harm that could arise from inaccurate or inconsistent screening advice. This is particularly relevant for health information seekers in North America, where access to up to date, evidence based guidance often intersects with individual risk factors and local policy considerations. Ultimately, AI tools may serve as valuable informational companions, but they should be used in conjunction with professional medical consultation to inform screening decisions. (Radiology study, University of Maryland)