Identifying Depression in Palliative Care Patients' Messages Using Large Language Models

Key Message : Large language models (LLMs) can identify depressive signals in patient-portal messages from individuals receiving palliative care with promising accuracy, supporting scalable, passive screening to enable timely psychosocial support.

Abstract: Background
Depression is common in palliative care and contributes to higher symptom burden, poor adherence, and lower quality of life. Yet, standardized screening is often underused due to workflow constraints. Patient-generated messages in electronic portals may provide an underutilized, real-world source of insight into emotional well-being. LLMs offer a potential means of detecting depressive language from these unstructured communications.

Objective
To evaluate the feasibility and accuracy of LLM-based methods for identifying depressive language in patient-initiated portal messages within a palliative care population.

Methods
We analyzed patient-initiated, English-language messages from a secure portal within the Emory Palliative Care Registry, encompassing 5,644 patients who received outpatient, emergency, or inpatient palliative care between January 1, 2022, and December 1, 2023. A stratified subset of 100 patients (50 with explicit depression mentions, 50 without) was independently annotated by two reviewers (agreement = 0.77) and cross-checked against patient-reported outcomes in the electronic health record. We evaluated several transformer-based models for binary depression classification: BERT, ClinicalBERT, MentalHealthBERT, and the LLaMA 3.1 model with few-shot and chain-of-thought prompting. Models were trained on 80% of the data with 5-fold cross-validation and tested on the remaining 20%.

Results
Few-shot LLaMA 3.1 and standard BERT achieved the highest performance (F1 = 0.77), outperforming ClinicalBERT (0.57) and MentalHealthBERT (0.72). Zero-shot LLaMA 3.1 achieved perfect precision (1.00) but low recall (0.40). Qualitative review suggested that LLaMA 3.1 more accurately captured contextual expressions of low mood (e.g., “tired of fighting,” “no reason to get up”) rather than only explicit diagnostic terms.

Conclusion
LLM-based models show early promise for passive depression screening using patient messages, offering a pathway toward continuous, low-burden monitoring of psychosocial distress in palliative care. Before integration into practice, prospective validation and fairness evaluation are needed to ensure clinical reliability, transparency, and equitable application across diverse patient populations.

References: N/A