Answering Patient Questions: AI Does Better Than Doctors

Eric Topol (4/28/23): When Patient Questions Are Answered With Higher Quality and Empathy by ChatGPT than Physicians

RE: JW Ayers et al. JAMA Intern Med 2023; doi:10.1001/jamainternmed.2023.1838 Open Access! Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum

Thanks to Jeff Lewis for sharing Eric Topol’s article. Here are some excerpts:

In JAMA Internal Medicine, a new report compared the quality and empathy of responses to patient questions for doctors vs ChatGPT, a generative AI model that has already been superseded by GPT4…

Researchers at UCSD used the Reddit social media platform (Reddit’s/AskDocs) to randomly selected 195 patient questions that had been answered by verified, volunteer physicians and also posed them to ChatGPT in an identical fashion. The answers were reviewed by a panel of 3 health care professionals blinded to whether the response was from a doctor or the chatbot…

The results were pretty striking, as shown in the graph below. For quality, the evaluators preferred the ChatGPT response 79% of the time; the quality score of good and very good was 79% for the chatbot vs 22% for physicians, a near 4-fold difference. That gap was even greater for empathy rating of the responses, nearly 10-fold higher favoring the chatbot, for proportion of empathetic or very empathic responses (45.1 vs 4.6%). As would be expected, the length of responses by doctors was significantly reduced (average of 52 vs 211 words, respectively). Of course, doctors have less time available and machines can generate such long-form output in seconds…

Several examples were presented, and I show one below that is representative. This actually happened to my mother-in-law several years ago and it was quite a frightening incident with extensive corneal injury.There were definite limitations of the study, since the questions were derived from a social medial platform, not from physicians who were actually involved in the care of the patient asking questions. And, of course, this was not comparing the additivity of a chatbot plus a physician..

My take: This study indicates that physicians/patients may benefit from leveraging chatbots to improve communication.  Artificial intelligence assistants can aid in drafting responses to patient questions. Though physicians, at this time, are more capable of providing patient-specific information. For more general questions, chatbots appear to do quite well. My personal office-related experience with ChatGPT includes drafting an appeal letter, providing educational material on encopresis, helping with a condolence letter, and researching a rare condition. Just two days ago (at the time of drafting this post), I asked ChatGPT to explain ulcerative colitis and treatments, including mesalamine and steroids, at an 8th grade reading level in Spanish. I found this was a good way to start a conversation with a family.

This study was covered in many news outlets including WSJ, CNN, Yahoo, Daily Mall…

Related blog posts:

Have you tried out ChatGPT?

There has been a lot written about ChatGPT (see some NY Times links below). Earlier this week, one of my sons placed the following prompt:

“Write me a funny poem about a doctor from Atlanta who deals with diarrhea all day long”

This is the ChatGPT response:

There once was a doctor from Atlanta
Whose patients all suffered from diarrhea
They came in by the dozens
Leaving behind a stench that was quite humbling
But this doctor was quite unflappable

He had a cure for every case
And a smile on his face
As he worked tirelessly
To help each patient feel less queasy

His office was filled with the sounds of relief
As patients left feeling much less grief
They praised his skill and his bedside manner
And promised to spread the word about this Atlanta diarrhea-banner

So if you’re in Atlanta and feeling quite ill
And your stomach is acting quite shrill
Don’t hesitate, don’t be shy
Just make your way to this doctor’s side
And say goodbye to that diarrhea for good!

NY Times Articles:

“It can extrapolate and take ideas from different contexts and merge them together.” The new chatbots do this with what seems like complete confidence. But they do not always tell the truth. Sometimes, they even fail at simple arithmetic. They blend fact with fiction. And as they continue to improve, people could use them to generate and spread untruths.

An example cited in this article:

My take: ChatGPT (& similar programs) is a huge technological advance. In educational settings, it could help explain complicated topics. Of course, there is a concern that it could undermine education if students do not go through the process of doing their own work. In healthcare, it is possible that these programs could advance patient education and help in the diagnostic process.

Another example (on 12/16/22) -this took less than 30 seconds: