Chat GPT does better than doctors at establishing a medical diagnosis. A study proves it!

Could Artificial Intelligence replace doctors? What sounds like a fictional scenario or a nightmare (depending on whether you’re a patient or a doctor) might just become reality.

Indeed, a study recent study carried out with around fifty doctors and medical interns, reveals that ChatGPT alone is capable, in 90% of cases, of making a reliable diagnosis. In contrast, doctors who used the software to make a diagnosis had a reliability score of 76% while the group who used only conventional resources had a median score of 74%.

One hour to establish 6 diagnoses

For this study, Stanford researchers offered doctors (and ChatGPT) 6 cases that have been used in medical research for decades. These cases had never been made public, so they were not included in the resources available to ChatGPT. Only one of the 6 files had been disclosed in advance with examples of correct and other false answers, to make it clear what was expected.

Doctors were then asked to make as many diagnoses as possible in an hour and half of them were allowed to use the chatbot in addition to traditional resources.

Many doctors do not know how to use AI to its best potential

The researchers were surprised to discover that the diagnosis success score was almost identical in the two groups of doctors since the doctors without help had an average score of 74% and the practitioners using the chatbot barely did better with 76%. In trying to understand why, they discovered that, due to lack of training, many doctors do not know how to exploit artificial intelligence to the best of its possibilities.

“Only a fraction of doctors realized that they could literally copy and paste the entire file into the chatbot and simply ask it to give a complete answer to the entire question” underlines Doctor Jonathan Chen, one of the authors of the study.

It was by doing this that ChatGPT was able to formulate 90% reliable diagnoses when the researchers tasked it, without the help of a doctor, with analyzing symptoms to determine an associated illness.

Source : Large Language Model Influence on Diagnostic ReasoningA Randomized Clinical Trial, Jama network, octobre 2024