Clinical Information Extraction with Large Language Models: A Case Study on Organ Procurement

Presentation Time: 03:30 PM - 03:45 PM

Abstract Keywords: Machine Learning, Information Extraction, Natural Language Processing
Primary Track: Foundations
Programmatic Theme: Clinical Research Informatics

Recent work has demonstrated that large language models (LLMs) are powerful tools for clinical information extraction from unstructured text. However, existing approaches have largely ignored the extraction of numeric information such as laboratory tests and vital signs. In this article, we present a case study on organ procurement that evaluates the ability of LLMs to extract numeric data from clinical text. We first describe our LLM-based approach, introducing a prompting strategy for numeric extraction and novel heuristics to combat hallucination. We validate our approach on a hand-annotated set of 298 notes, demonstrating that it has high accuracy, precision and recall. We then highlight the value of our approach for downstream data analysis using a corpus of 43,719 notes on 14,342 potential organ donors. This case study is a key component of an ongoing collaboration that aims to make data on organ procurement publicly available for informatics research.

Speaker(s):
Hammaad Adam, MS
Massachusetts Institute of Technology

Author(s):

Boosting Social Determinants of Health Extraction with Semantic Knowledge Augmented Large Language Model

Presentation Time: 03:45 PM - 04:00 PM

Abstract Keywords: Delivering Health Information and Knowledge to the Public, Information Extraction, Large Language Models (LLMs), Information Retrieval
Primary Track: Applications
Programmatic Theme: Public Health Informatics

Social determinants of health (SDoH) significantly impacts health outcomes and contributes to perpetuating health disparities across healthcare applications. However, automatic extraction of SDoH information from Electronic Health Records (EHRs) is challenging due to the unstructured nature of clinical narratives that contain SDoH related information. Recent advances in Large Language Models (LLMs) have shown great promise for automated SDoH extraction. However, their performance suffers for the imbalanced SDoH categories due to the data scarcity issues. To address this, we propose an innovative approach that augments LLMs with semantic knowledge obtained from the Unified Medical Language Systems (UMLS). This strategy enriches the feature representations of imbalanced SDoH classes, leading to accurate SDoH extraction. More specifically, our proposed data augmentation strategy generates semantically enriched clinical narratives at the LLM pre-finetuning stage. This approach enables the LLM to better adapt to the target data and leads to a good initialization for the finetuning stage. Through extensive experiments using publicly available MIMIC-SDoH data, the proposed approach demonstrates significant improvement in results for the SDoH extraction, especially for the imbalanced classes.

Speaker(s):
Kishlay Jha, PhD
University of Iowa

Author(s):

A Large Language Model Outperforms Other Computational Approaches to the High-Throughput Phenotyping of Physician Notes

Presentation Time: 04:00 PM - 04:15 PM

Abstract Keywords: Information Retrieval, Terminology Systems, Controlled Terminologies, Ontologies, and Vocabularies, Natural Language Processing, Documentation Burden
Primary Track: Applications
Programmatic Theme: Clinical Research Informatics

High-throughput phenotyping, the automated mapping of patient signs and symptoms to standardized ontology concepts, is essential to gaining value from electronic health records (EHR) in the support of precision medicine. Despite technological advances, high-throughput phenotyping remains a challenge. This study compares three computational approaches to high-throughput phenotyping: a Large Language Model (LLM) incorporating generative AI, a Natural Language Processing (NLP) approach utilizing deep learning for span categorization, and a hybrid approach combining word vectors with machine learning. The approach that implemented GPT-4 (a Large Language Model) demonstrated superior performance, suggesting that Large Language Models are poised to be the preferred method for high-throughput phenotyping of physician notes.

Speaker(s):
Syed Munzir, Student
University of Illinois at Chicago

Daniel Hier, MD
Retired

Author(s):

Enhancing Early Detection of Cognitive Decline in the Elderly through Ensemble of NLP Techniques: A Comparative Study Utilizing Large Language Models in Clinical Notes

Presentation Time: 04:15 PM - 04:30 PM

Abstract Keywords: Natural Language Processing, Large Language Models (LLMs), Machine Learning, Deep Learning
Primary Track: Applications
Programmatic Theme: Clinical Informatics

This study aims to leverages large language models (LLMs) in secure clouds for a pioneering exploration of EHR note analysis for cognitive decline detection. Our findings indicate that LLMs and traditional models exhibit diverse error profiles. The ensemble of LLMs and locally trained machine learning models on EHR data was found to be complementary, enhancing performance and improving diagnostic accuracy, with a marked improvement in precision (from a 70%-79% range to above 90%).

Speaker(s):
Xinsong Du, Ph.D.
Brigham and Women's Hospital/Harvard Medical School

Author(s):
Xinsong Du, Ph.D. - Brigham and Women's Hospital/Harvard Medical School; John Laurentiev, MS; Joseph Plasek, PhD - Mass General Brigham; Ya-Wen Chuang, MD, MPH - BRIGHAM AND WOMEN'S HOSPITAL; Liqin Wang, PhD - Brigham and Women's Hospital; Surabhi Datta, PhD; Hunki Paek, PhD; Lin Bin, MS - Intelligence Medical Objects; Qiang Wei - The University of Texas Health Science at Houston; Xiaoyan Wang, PhD in Biomedical Informatics - MelaxTech; Jingqi Wang - Melax Technologies, Inc; Hao Ding, Ph.D. - Intelligent Medical Objects; Frank Manion, PhD - Intelligent Medical Objects; Jingcheng Du, Ph.D. - Melax Tech; Li Zhou, MD, PhD, FACMI, FIAHSI, FAMIA - Brigham and Women's Hospital, Harvard Medical School;

Extraction of Normalized Symptom Mentions From Clinical Narratives Using Large Language Models

Presentation Time: 04:30 PM - 04:45 PM

Abstract Keywords: Natural Language Processing, Large Language Models (LLMs), Machine Learning
Primary Track: Applications
Programmatic Theme: Clinical Research Informatics

Symptoms, or subjective experiences of patients which can indicate underlying pathology, are important for guiding clinician decision-making and revealing patient wellbeing. However, they are difficult to study because information is primarily found in clinical free text, not in structured electronic health record fields. This study finds that large language models (LLMs) can extract several common symptom concepts from clinical narratives, using an approach of including clarifying information in the prompt, few-shot examples, and chain-of-thought-prompting. This approach is compared to symptom-specific machine learning classifiers based on clinical concepts mapped from free text. For most symptom concepts, the LLM performs better and achieves a higher F1-score, likely by leveraging context important for the symptom normalization task. Unlocking information about symptom concepts from clinical narratives has potential to improve healthcare workflows and facilitate a broad range of research agendas.

Speaker(s):
Afia Khan, BA
University of Chicago

Author(s):

Utilizing GPT-4 to determine reasons for missed follow-up colonoscopy following abnormal non-invasive colorectal cancer screening

Presentation Time: 04:45 PM - 05:00 PM

Abstract Keywords: Large Language Models (LLMs), Natural Language Processing, Population Health
Primary Track: Applications
Programmatic Theme: Clinical Informatics

We sought to investigate the use of GPT-4 to determine reasons for missed follow-up colonoscopy among patients who have had abnormal non-invasive colorectal cancer screening tests. We show that, using de-identified clinical notes, GPT-4 can identify and extract reasons for missed follow-up colonoscopy with 89.3% accuracy. We also explore the key reasons identified for missing follow-up colonoscopy and discuss the benefits and limitations of utilizing large language models for quality improvement in healthcare.

Speaker(s):
Christopher Williams, MB BChir
UCSF

Author(s):
Daniel Broman, BSIE - UCSF; Urmimala Sarkar, MD MPH - UCSF; Julia Adler-Milstein, PhD - UCSF School of Medicine; Lisa Rotenstein, MD, MBA, MSc - UCSF;