Eric Topol (4/28/23): When Patient Questions Are Answered With Higher Quality and Empathy by ChatGPT than Physicians
RE: JW Ayers et al. JAMA Intern Med 2023; doi:10.1001/jamainternmed.2023.1838 Open Access! Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum
Thanks to Jeff Lewis for sharing Eric Topol’s article. Here are some excerpts:
In JAMA Internal Medicine, a new report compared the quality and empathy of responses to patient questions for doctors vs ChatGPT, a generative AI model that has already been superseded by GPT4…
Researchers at UCSD used the Reddit social media platform (Reddit’s/AskDocs) to randomly selected 195 patient questions that had been answered by verified, volunteer physicians and also posed them to ChatGPT in an identical fashion. The answers were reviewed by a panel of 3 health care professionals blinded to whether the response was from a doctor or the chatbot…
The results were pretty striking, as shown in the graph below. For quality, the evaluators preferred the ChatGPT response 79% of the time; the quality score of good and very good was 79% for the chatbot vs 22% for physicians, a near 4-fold difference. That gap was even greater for empathy rating of the responses, nearly 10-fold higher favoring the chatbot, for proportion of empathetic or very empathic responses (45.1 vs 4.6%). As would be expected, the length of responses by doctors was significantly reduced (average of 52 vs 211 words, respectively). Of course, doctors have less time available and machines can generate such long-form output in seconds…
Several examples were presented, and I show one below that is representative. This actually happened to my mother-in-law several years ago and it was quite a frightening incident with extensive corneal injury.There were definite limitations of the study, since the questions were derived from a social medial platform, not from physicians who were actually involved in the care of the patient asking questions. And, of course, this was not comparing the additivity of a chatbot plus a physician..
My take: This study indicates that physicians/patients may benefit from leveraging chatbots to improve communication. Artificial intelligence assistants can aid in drafting responses to patient questions. Though physicians, at this time, are more capable of providing patient-specific information. For more general questions, chatbots appear to do quite well. My personal office-related experience with ChatGPT includes drafting an appeal letter, providing educational material on encopresis, helping with a condolence letter, and researching a rare condition. Just two days ago (at the time of drafting this post), I asked ChatGPT to explain ulcerative colitis and treatments, including mesalamine and steroids, at an 8th grade reading level in Spanish. I found this was a good way to start a conversation with a family.
This study was covered in many news outlets including WSJ, CNN, Yahoo, Daily Mall…
Related blog posts:
gutsandgrowth 