Does AI Evaluate Patients' Health More Accurately Than Doctors?

The study was conducted at Cedars-Sinai Connect, a virtual urgent care clinic in Los Angeles focused on primary and urgent care, where an AI system was recently implemented. The retrospective cohort study compared diagnoses and treatment recommendations made by the AI and by clinic physicians. Doctors had access to the AI’s suggestions but could choose whether or not to review them.

Online Consultations

Professor Zeltzer, whose findings were published in the Annals of Internal Medicine and presented at the annual conference of the American College of Physicians (ACP), explained the research steps in detail.

Experts thoroughly examined 461 consultations with adult patients who contacted the online clinic with common health issues—seeking consultations for respiratory, urinary, vaginal, eye, or dental problems.

The patients’ conditions were first assessed by the AI, which conducted an intake interview, reviewed the patients’ medical records, and then recommended specific therapies. The patients were then consulted by a physician via video call. A panel of four medical experts with at least 10 years of experience reviewed both the doctors’ and AI’s recommendations and evaluated each based on medical history, data gathered during the visit, and transcripts of the video consultation. Recommendations were rated as optimal, reasonable, inappropriate, or potentially harmful.

How AI Treats

The algorithm, trained on millions of medical records, determined diagnoses and proposed treatment plans that included prescriptions, tests, and further recommendations. A key aspect was the reliability of AI recommendations. Therefore, the algorithm only offered treatment advice when the confidence level was high—otherwise, it withheld recommendations (about 1 in 5 cases).

So how did the quality of AI recommendations compare to those of physicians? The expert panel rated AI’s recommendations as optimal in 77% of cases, compared to only 67% for physicians.

At the other end of the spectrum, AI recommendations were deemed potentially harmful less often than those made by doctors (2.8% of AI cases vs. 4.6% of physician cases). In 68% of consultations, AI and physicians received the same score. In 21% of cases, the AI scored higher; in 11%, the doctor’s decision was considered better.

Collaboration Can Bring Benefits

Professor Zeltzer’s study is unique in that the AI was tested in a real clinical setting with real patients. The findings suggest that AI adheres more strictly to guidelines and processes a broader and more comprehensive range of patient information from medical records. It also appears to better identify symptoms that may indicate serious conditions.

Physicians, on the other hand, are more flexible and have the advantage of assessing the patient’s current clinical state. They were better at adapting recommendations based on changing information during the consultation.

A limitation of the study is that it’s unknown which physicians reviewed the AI recommendations or to what extent they relied on them. Only the accuracy of the recommendations was evaluated—not their influence on physician decisions. Still, the study suggests that AI can provide diagnostic and therapeutic advice that is often more accurate than that of physicians and can highlight information that refines diagnoses.

Because AI appears to be a promising aid in therapeutic decision-making, researchers continue working on its effective integration into diagnostic and treatment processes. The goal is to achieve an optimal balance between human expertise and AI in medicine so that the collaboration between doctors and AI leads to the fastest and most efficient recovery of patients.

Editorial Team, Medscope.pro

Sources:

1. Zeltzer D., Kugler Z., Hayat L. et al.: Comparison of Initial Artificial Intelligence (AI) and Final Physician Recommendations in AI-Assisted Virtual Urgent Care Visits. Ann Intern Med. April 2025; 178(4):498-506, doi: 10.7326/ANNALS-24-03283.

2. Diagnoses and Treatment Recommendations Given by AI were More Accurate than those of Physicians. Digital Health News, April 17, 2025. Available at: www.digitalhealthnews.eu/research/7429-diagnoses-and-treatment-recommendations-given-by-ai-were-more-accurate-than-those-of-physician