Natural Language Processing for Extracting Social Determinants of Health from Unstructured Clinical Notes: Impact on Patient Risk Stratification

Dr. Sophie Dubois; Prof. Min-Joon Kim; Dr. Fatima Zahraoui

Abstract

Background: Social Determinants of Health (SDOH) significantly impact disease presentation, progression, and treatment outcomes. However, these crucial factors are often embedded in unstructured clinical notes within Electronic Health Records (EHRs), making them difficult to access and integrate into risk stratification models. This study aims to develop and validate a natural language processing (NLP) model to automatically extract key SDOH information and assess its utility in enhancing existing predictive models for patient risk. Methods: A retrospective analysis was conducted using 100,000 anonymized clinical notes from a diverse patient cohort. A custom NLP pipeline, combining rule-based methods and transformer-based deep learning (BERT), was developed to identify and extract categories such as housing instability, food insecurity, transportation barriers, and social support. Extracted SDOH data were then integrated into a conventional risk prediction model for 30-day hospital readmission, comparing its performance against a model without SDOH data. Results: The NLP model achieved an F1-score of 0.89 for identifying composite SDOH, with precision ranging from 0.85 (transportation) to 0.92 (housing). Integrating extracted SDOH features improved the AUC of the 30-day readmission prediction model from 0.78 (95% CI: 0.76-0.80) to 0.83 (95% CI: 0.81-0.85), representing a statistically significant improvement (p < 0.001). Conclusion: NLP offers a powerful and scalable solution for extracting critical SDOH from unstructured clinical text, significantly enhancing the accuracy of patient risk stratification and enabling more holistic care planning.

Keywords: NLP, social determinants of health, EHR, risk stratification, clinical notes

Social determinants of health are increasingly recognized as fundamental drivers of health disparities and outcomes, yet their systematic capture and integration into clinical decision-making remain challenging due to their qualitative nature and presence in unstructured text. Automating the extraction of this vital information through advanced Natural Language Processing (NLP) can revolutionize patient risk stratification, facilitate precision medicine, and inform targeted interventions to address health inequities. This research directly tackles a critical bottleneck in leveraging holistic patient data.

The aim of the study.

To develop and validate an NLP-based methodology for the automated extraction of pivotal social determinants of health from unstructured clinical notes within electronic health records, and subsequently to evaluate the impact of integrating these extracted features on the performance of a 30-day hospital readmission prediction model.

Material and methods.

This retrospective study utilized a de-identified dataset of 100,000 clinical notes (admission notes, discharge summaries, social work assessments) from patients admitted to a large urban hospital. A hybrid NLP approach was employed, combining rule-based text matching and a fine-tuned BERT (Bidirectional Encoder Representations from Transformers) model for named entity recognition and relation extraction. Key SDOH categories included: housing status (stable, unstable, homeless), food security (secure, insecure), transportation access (present, barrier), and social support (strong, moderate, weak, isolated). A randomly selected subset of 5,000 notes was manually annotated by two independent clinical experts to serve as the gold standard for NLP model training and validation. The extracted SDOH features were then added to a baseline logistic regression model predicting 30-day hospital readmission, which originally utilized only structured demographic and clinical data (e.g., age, comorbidities, previous admissions). Model performance was assessed using AUC, F1-score, precision, and recall, with statistical comparisons via Delong's test.

The custom NLP pipeline achieved an overall F1-score of 0.89 for the composite extraction of SDOH categories. Performance varied slightly by category: housing status (Precision: 0.92, Recall: 0.90, F1: 0.91), food security (Precision: 0.88, Recall: 0.86, F1: 0.87), transportation barriers (Precision: 0.85, Recall: 0.82, F1: 0.83), and social support (Precision: 0.90, Recall: 0.87, F1: 0.88). When integrated into the 30-day hospital readmission prediction model, the AUC significantly improved from 0.78 (95% CI: 0.76-0.80) for the baseline model to 0.83 (95% CI: 0.81-0.85) for the SDOH-enhanced model (p < 0.001). This represented a 6.4% relative increase in predictive power. The adjusted odds ratio for readmission given housing instability, extracted by NLP, was 1.7 (95% CI: 1.5-1.9, p<0.001).

This study conclusively demonstrates the substantial value of employing advanced NLP techniques to systematically extract social determinants of health from the verbose, unstructured content of clinical notes. The integration of these NLP-derived SDOH features significantly boosted the predictive accuracy of a critical clinical outcome model (30-day hospital readmission). By making latent SDOH data actionable, this methodology promises to empower clinicians with a more comprehensive understanding of patient risk, paving the way for more equitable, personalized, and effective healthcare interventions. Future work will focus on prospective validation and integration into clinical decision support systems.

1. Byrd JC, et al. Social Determinants of Health in Clinical Practice. Med Clin North Am. 2019;103(6):1001-1014.

2. Wang Y, et al. Natural Language Processing for Social Determinants of Health: A Scoping Review. J Biomed Inform. 2021;119:103774.

3. Devlin J, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL HLT. 2019;1:4171-4186.

4. Weiner M, et al. Automated extraction of social determinants of health from clinical notes for improved chronic disease management. AMIA Joint Summits Transl Sci Proc. 2020;2020:645-654.

5. D'Souza J, et al. Improving Risk Prediction with Social Determinants of Health Extracted from Clinical Notes. J Am Med Inform Assoc. 2020;27(10):1559-1568.