Development of a Flexible Chain of Thought Framework for Automated Routing of Patient Portal Messages

Presentation Time: 10:30 AM - 10:45 AM

Abstract Keywords: Large Language Models (LLMs), Documentation Burden, Natural Language Processing, Workflow
Primary Track: Applications

The increase in utilization of patient portal messages has imposed a considerable burden on healthcare providers, contributing to an increased incidence of provider burnout. This study introduces a framework for leveraging Large Language Models (LLMs) and Chain-of-Thought (CoT) prompting in order to automatically categorize and route messages to their appropriate location. The modeling framework, which utilizes gold standard annotations from triage nurses, not only facilitates the dynamic adaptation of the model to evolving healthcare workflows and emerging edge-case scenarios, but also significantly improves the model's classification accuracy compared to traditional zero-shot methods. In addition, the framework allows for flexibility in its task and continuous improvement via annotation of exemplar messages. The model is able to accurately categorize messages in an automated fashion, which has potential to dramatically ease the burden on providers and provide faster and safer responses to patients. This framework can also be readily extended to work in a variety of clinical and documentation settings.

Speaker(s):
Michael Gao, BS
Duke University

Author(s):
Ricardo Henao, PhD - Duke University; Suresh Balu, MS, MBA - Duke University; Kartik Pejavara, BS - Duke University;

Behavior Shifts in Patient Portal Usage During and After Policy Changes Around Test Result Delivery and Notification

Presentation Time: 10:45 AM - 11:00 AM

Abstract Keywords: Behavioral Change, Patient Engagement and Preferences, Telemedicine, Population Health, Delivering Health Information and Knowledge to the Public
Primary Track: Policy
Programmatic Theme: Consumer Health Informatics

Because of the 21st Century Cures Act, many health systems now release all test results into patient portals immediately. To investigate if changes in access to test results shifted patient portal usage, we used data from the electronic health record to evaluate how patients behaved after this policy change and a subsequent policy adjustment requiring patients to opt in for notifications about new test results. We found that following institutional compliance with the Cures Act, proportions of patients who scheduled a new appointment and messaged their clinician after accessing a new test result increased, both by 4.5%. After removing automatic notifications of new results, the proportion of patients who scheduled a new appointment increased by 2.1%, and the proportion of patients who had telemedicine encounters decreased by 0.8%. Our work identified changes in patient behavior that track how policy changes map to burden for clinicians and information-seeking behavior in patients.

Speaker(s):
Uday Suresh, MS
Vanderbilt University Department of Biomedical Informatics

Author(s):
Uday Suresh, MS - Vanderbilt University Department of Biomedical Informatics; Bryan Steitz, PhD - Vanderbilt University Medical Center; S. Trent Rosenbloom, MD, MPH, FACMI, FAMIA - Vanderbilt University Medical Center Dept of Biomedical Informatics; Kevin Griffith; Jessica Ancker, MPH, PhD, FACMI - Vanderbilt University Medical Center;

Improving Emergency Department Visit Risk Prediction: Exploring the Operational Utility of Applied Patient Portal Messages

Presentation Time: 11:00 AM - 11:15 AM

Abstract Keywords: Large Language Models (LLMs), Natural Language Processing, Clinical Decision Support, Patient / Person Generated Health Data (Patient Reported Outcomes), Mobile Health, Population Health
Primary Track: Applications
Programmatic Theme: Clinical Informatics

Patient portal messages represent a unique source of clinical data due to how they represent the voice of the patient, provide a glimpse into care delivery between episodic synchronous appointments, and capture variations in patient behavior and health literacy. There is little understanding of how to best apply modern natural language processing (NLP) approaches, such as large, pre-trained language models (LLMs), to patient messages. In this study, we aim to explore different approaches in incorporating patient messages into an existing ED visit risk prediction model currently deployed at Stanford Health Care. With the addition of patient message frequencies to the baseline we were able to achieve an improved AUC of .83 and a significant jump in the F1 score. In future work, we aim to build upon these findings and further test combination models to incorporate features around patient message content, in addition to message frequencies.

Speaker(s):
Hanna Kiani, MHA
Stanford Health Care

Author(s):
Sohaib Hassan, BA Genetics - Stanford University; Sohaib Hassan, PhD Biomedical Data Science (in-training) - Stanford Department of Biomedical Data Science;

Towards Optimizing LLM Use in Healthcare: Identifying Patient Questions in MyChart Messages

Presentation Time: 11:15 AM - 11:30 AM

Abstract Keywords: Healthcare Quality, Documentation Burden, Natural Language Processing, Deep Learning, Informatics Implementation, Personal Health Informatics, Delivering Health Information and Knowledge to the Public, Information Extraction
Primary Track: Applications
Programmatic Theme: Clinical Informatics

The volume of patient-provider messages is on the rise, and Large Language Models (LLMs) can potentially streamline the clinical messaging process, but their success hinges on triaging messages they can optimally address. In this study, we analyzed Electronic Health Records with over 4 million messages exchanged between patients and providers to characterize the utility of using LLMs for messages containing knowledge questions. We implemented a rule-based Syntactic Question Detector as a triage tool, and we evaluated it on 500 messages. The interrater reliability metrics and comparison with LLMs show the difficulty of detecting questions due to the informal text and implicit requests. Our results show that 25% of MyChart messages with questions do not have a response from the clinical team. This paper provides insights into the challenges of real-world data, highlights the importance and non-triviality of detecting questions, and suggests a pipeline for LLM use in healthcare.

Speaker(s):
Michael Hogarth, MD
University of California at San Diego

Author(s):
Akhila Chekuri, MS - University of California San Diego; Armaan S. Johal, N/A - University of California San Diego; Matthew Allen, BS - University of California, San Diego; John W. Ayers, PhD, MA - University of California San Diego; Michael Hogarth, MD - University of California at San Diego; Emilia Farcas, PhD - University of California San Diego;

Does Transparency Promote Engagement? Cancer Patients’ Access of Electronic Medical Records Before and After the Information Blocking Rule

Presentation Time: 11:30 AM - 11:45 AM

Abstract Keywords: Patient Engagement and Preferences, Evaluation, Personal Health Informatics
Primary Track: Policy
Programmatic Theme: Consumer Health Informatics

Access to clinical information is critical to support patient engagement. The 21st Century Cures Act grants patients immediate electronic access to their full medical records. To assess the potential impact of this transparency provision, we conducted a retrospective study in a large cancer center in New York City, focusing on clinically active patients’ accessing health information shared via the patient portal. We identified a significant increase (14%) in the number of pathology and radiology reports read by patients after the implementation of immediate release of reports. No changes were found in the rates of account creation or logins. Our results suggest that oncology patients show strong, consistent interest in their clinical data, with many taking advantage of the full electronic access granted by Cures. These findings shed new light on this legislation’s impact on patient engagement and access to clinical data.

Speaker(s):
Fernanda Polubriaginof, MD PhD
Memorial Sloan Kettering Cancer Center

Author(s):
Fernanda Polubriaginof, MD PhD - Memorial Sloan Kettering Cancer Center; SUSAN CHIMONAS, PhD - Memorial Sloan Kettering Cancer Center; Allison Lipitz-Snyderman, PhD - Memorial Sloan Kettering Cancer Center; Zoe Spiegelhoff; Kenneth Seier, MS - Memorial Sloan Kettering Cancer Center; Charles White, MS - Memorial Sloan Kettering Cancer Center; Joshua Jorvina, BSN, RN, OCN; Gilad Kuperman, MD, PhD - Memorial Sloan Kettering Cancer Center;

Critical access hospital adoption of advanced patient engagement functions

Presentation Time: 11:45 AM - 12:00 PM

Abstract Keywords: Patient Engagement and Preferences, Interoperability and Health Information Exchange, Legal, Ethical, Social and Regulatory Issues
Primary Track: Policy
Programmatic Theme: Consumer Health Informatics

Critical access hospitals have historically lagged other hospitals in their adoption of advanced functionalities, including patient engagement functions. We analyzed AHA survey data from 2014, 2018, and 2023 to assess how CAH adoption gaps have evolved over time. The gap in core patient engagement functions has closed, however new adoption gaps have emerged in functions focused on FHIR and interoperability of patient-generated health data.

Speaker(s):
Nate Apathy, PhD
University of Maryland

Author(s):
Jordan Everson - Office of the National Coordinator for Health Information Technology; Paige Nong, PhD - University of Minnesota; A J Holmgren, PhD - University of California, San Francisco; Julia Adler-Milstein, PhD - UCSF School of Medicine;