Scientists from AnsibleHealth demonstrated that the ChatGPT bot was able to achieve the minimum passing scores for the USMLE exam to obtain a medical license in the United States. The research was published in the journal PLOS Digital Health.
The USMLE is a series of three exams required to obtain a medical license in the United States. The questions cover most medical disciplines, from biochemistry to diagnostics and bioethics.
Scientists tested ChatGPT on 350 of 376 USMLE public questions as of June 2022.
The bot scored between 52.4% and 75.0% on three USMLE exams. The pass threshold is approximately 60% each year. In addition, the bot was able to produce non-obvious and clinically relevant output in 88.9% of its responses. ChatGPT outperformed PubMedGPT, a similar model trained solely on biomedical literature, scoring 50.8% on the USMLE exam.
The authors noted that their results support the potential of ChatGPT to improve medical education and clinical practice. Doctors at AnsibleHealth are already using ChatGPT to rewrite complex reports in language that patients can understand.