Comparing prognostic performance and reasoning between physicians and large language models

Key Message : This work compares the performance of practicing physicians and large language models (LLMs) when prognosticating 6-month mortality for critically ill adults, and we found similar performance. As the interest in LLMs continues to grow, it is essential that their role in clinical practice and the ethical implications are carefully considered.

Abstract:

Background: Physicians routinely prognosticate to guide shared decision making, particularly when caring for patients with critical illnesses (1). Yet, these physician estimates are prone to inaccuracy and uncertainty (2,3). Artificial intelligence, including large language models (LLMs), shows promise in supporting or improving prognostication (4). However, the performance of contemporary LLMs in prognosticating for the heterogeneous population of critically ill patients remains poorly understood.

Objective: To compare the performance and reasoning of LLMs and physicians when predicting 6-month mortality for hospitalized, critically ill adults.

Methods: We randomly selected 100 patient admissions with ICU stays from MIMIC-IV, a publicly available, single-center dataset of hospitalizations. For each case, 4 contemporary LLMs and 7 practicing physicians were asked to use the discharge summary and demographics to (1) predict 6-month mortality (yes/no) and (2) provide their reasoning. Physicians also rated their confidence level (0—100). We assessed prognostic performance using accuracy, sensitivity, specificity and used inductive, qualitative content analysis to characterize reasoning.

Results: Mean physician accuracy was 70.1% (95% CI 63.1—76.3%), with sensitivity of 59.7% and specificity of 80.6%. The top-performing LLM (OpenAI o4-mini) achieved 78.0% accuracy (95% CI 70.0—86.0%), with sensitivity of 80.0% and specificity of 76.0%. Individual physician accuracies ranged from 63.0% to 77.0%; LLMs ranged from 70.0% to 78.0%. When physicians’ confidence was low (lowest quartile), their mean accuracy dropped to 53.2% (95% CI 45.8–60.2%), while o4-mini’s was 57.9% (95% CI 36.8–78.9%). Qualitative analysis revealed similar patterns in LLM and physician reasoning responses.

Conclusion/Relevance: Physicians and LLMs demonstrate moderate prognostic performance when predicting 6-month mortality in critically ill adults with comparable levels of accuracy. Our qualitative findings also suggest LLMs can emulate physicians’ complex clinical reasoning. As LLMs approach physician-level performance, careful consideration of their role in clinical practice and ethical deployment is essential.

References: 1. White DB, Ernecoff N, Buddadhumaruk P, et al. Prevalence of and Factors Related to Discordance About Prognosis Between Physicians and Surrogate Decision Makers of Critically Ill Patients. JAMA. 2016 May 17;315(19):2086-94. doi: 10.1001/jama.2016.5351. PMID: 27187301.

2. Detsky ME, Harhay MO, Bayard DF, et al. Discriminative Accuracy of Physician and Nurse Predictions for Survival and Functional Outcomes 6 Months After an ICU Admission. JAMA. 2017 Jun 6;317(21):2187-2195. doi: 10.1001/jama.2017.4078. PMID: 28528347; PMCID: PMC5710341.

3. Ramos JGR, Dias RD, Passos RDH, Batista PBP, Forte DN. Prognostication in urgent intensive care unit referrals: a cohort study. BMJ Support Palliat Care. 2020 Mar;10(1):118-121. doi: 10.1136/bmjspcare-2018-001567. Epub 2018 Aug 31. PMID: 30171040.

4. Chung P, Fong CT, Walters AM, Aghaeepour N, Yetisgen M, et al. Large Language Model Capabilities in Perioperative Risk Prediction and Prognostication. JAMA Surg. 2024 Aug 1;159(8):928-937. doi: 10.1001/jamasurg.2024.1621. PMID: 38837145; PMCID: PMC11154375.