Using a Large Language Model to Assess Documentation of Electronically Reported Symptoms in Oncology

Key Message : In a retrospective study of 5,049 visit notes from 552 patients, a large language model accurately identified 10 common symptoms. Among symptoms reported electronically by patients, only 31-71% were addressed in visit notes. Findings suggest a gap between patient experiences and clinical documentation, highlighting the importance of electronic patient-reported outcomes.

Abstract:

Background: Oncology practices increasingly collect electronic patient-reported outcomes (ePROs). While ePROs provide valuable insights, it is unknown whether electronically-reported symptoms are addressed during clinic visits. Large language models (LLMs) present a novel opportunity to systematically compare ePROs to symptoms documented by clinicians.

Objectives: To assess LLM performance identifying symptoms in visit notes and evaluate concordance between clinician-documented symptoms and ePROs.

Methods: In this retrospective study, we analyzed records of adults with an estimated prognosis < 1 year (1) evaluated in gastrointestinal and gynecologic oncology clinics from January 2022-July 2023. We manually annotated 12 symptoms in a training dataset of 500 notes (gold standard), then prompted a HIPAA-compliant LLM (GPT4DFCI 4o) to identify these symptoms. To assess LLM performance compared to the gold standard, we calculated model F1 scores (goal ≥0.70) for each symptom. We then used the LLM to identify symptoms in notes written within four days of an ePRO submission and calculated sensitivity and specificity of notes to ePRO symptoms by severity.

Results: Of 772 eligible patients, 552 submitted an ePRO during the study period. Notes from 5,049 visits were included in the analysis. Model F1 scores ranged from 0.60 (trouble drinking) to 0.90 (numbness/tingling); scores were 0.70-0.79 for four symptoms and ≥0.80 for six symptoms. Among the 10 accurately identified symptoms, visit note sensitivity compared to ePROs ranged from 0.31 (anxiety) to 0.71 (pain); specificity ranged from 0.65 (fatigue) to 0.95 (dyspnea). When only severe symptoms were considered, sensitivity ranged from 0.57 (anxiety) to 0.94 (pain).

Conclusions: The LLM accurately identified most symptoms. Notes reliably reflected ePRO reports of pain and absence of dyspnea; more severe symptoms were more often documented. There is a gap between patient reports and clinician documentation of anxiety, highlighting a potentially unmet need and underscoring the importance of ePROs when characterizing symptoms.

References: 1. Cotner CE, Tramontano AC, Post A, et al. Using Oncology Treatment Pathway Data to Evaluate Serious Illness Communication, Care Utilization, and End-of-Life Care for Patients With Cancer. JCO Oncol Pract. 2025 Apr;21(4):535-543. doi: 10.1200/OP.24.00311. Epub 2024 Sep 30. PMID: 39348633.