Utilization and Impact of Artificial Intelligence-Generated Draft Replies to Patient Messages in Pediatrics

2025 Clinical Informatics Conference 25x5 Presentation

Presentation Time: 08:00 AM - 08:15 AM

Abstract Keywords: Artificial Intelligence/Machine Learning, Driving Digital Equity, Usability and Measuring User Experience, Documentation Burden
Primary Track: Documentation Burden, Clinician Well-Being and Care Outcomes
Programmatic Theme: Usability, Efficiency, and User Experience

AI-generated draft replies to patient messages have not been studied in pediatric clinical settings, raising concerns about the feature’s acceptability and applicability to this context. In this single-site cohort study, users from both pediatric- and adult-facing specialties were given access to AI-generated drafts. Pediatric providers reported significant reduction in task load associated with responding to patient messages and recommended the tool more highly than adult providers despite overall lower utilization of generated drafts.

Speaker:
April Liang, MD
Stanford University

Authors:
Shivam Vedak, MD, MBA - Stanford Medicine; Alex Dussaq, MD, PhD - Stanford; Dong-han Yao, MD - Stanford; Joshua Villarreal, MD - Stanford Healthcare; Sijo Thomas, RN, MSN, MBA, PMP - Stanford Children's Health; Nicholas Chen, BA - Stanford Children's Health; Tanya Townsend, MSMI - Stanford Children's Health; Natalie Pageler, MD, MEd - Stanford University; Keith Morse, MD - Stanford University School of Medicine;

---

Adoption and Utility: Evaluation of Usage Rate and Editing Overlap for Artificial Intelligence-Drafted Replies to Patient Messages

2025 Clinical Informatics Conference 25x5 Presentation

Presentation Time: 08:15 AM - 08:30 AM

Abstract Keywords: Artificial Intelligence/Machine Learning, EHR Implementation and Optimization, Communication Strategies, Ambient documentation
Primary Track: AI and Care Outcomes
Programmatic Theme: Emerging Technology and Technical Infrastructure

This session will provide insight into the implementation and evaluation of Generative Artificial Intelligence (AI) technology to respond to patient portal messages. Attendees will learn about the high variation in usage rates and editing efforts, prompting a discussion of best practices for implementation of these technologies in attendees’ own clinical settings. They will also learn data analysis and visualization strategies they can apply when implementing and evaluating the utility of AI tools in clinical practice.

Speaker:
ASAF Hanish, MPH
Penn Medicine

Authors:
Asaf Hanish, MPH - University of Pennsylvania Health System; Aidan Crowley, BS - University of Pennsylvania; Michael Becker, BS - University of Pennsylvania Health System; Jason Lubken, BS - University of Pennsylvania Health System; Jeffrey Moon, MD MPH - Hospital of the University of Pennsylvania; Susan Regli, PhD - University of Pennsylvania Health System;

---

Enhancing Patient Communication: The Impact of LLMs on Care Team Messaging Workflows

2025 Clinical Informatics Conference 25x5 Presentation

Presentation Time: 08:30 AM - 08:45 AM

Abstract Keywords: Artificial Intelligence/Machine Learning, Clinical Process Automation, Patient Safety
Primary Track: AI and Care Outcomes
Programmatic Theme: Usability, Efficiency, and User Experience

The deployment of AI-generated draft replies in healthcare settings has introduced new dynamics in how medical assistants (MAs), nurses (RNs), and physicians (MDs) interact with patient messages. We performed inductive coding of messages sent with and without access to draft replies to investigate their influence on scope creep and error rates in message handling by MAs and found that scope creep was rare and did not appear to be adversely affected by the use of AI-generated draft messages. We also explored the utility of fine-tuned LLMs for the automated triage of incoming messages, which would have the potential to redirect incoming patient messages with clinical questions to the appropriate role with the appropriate urgency. We found that even small open-source LLMs such as Llama-3-8B was able to achieve high accuracy when compared to human labels. These results underscore the potential for increased automation through the use of LLMs in clinical settings.

Speaker:
Stephen Ma, MD, PhD
Stanford University School of Medicine

Authors:
Tom Savage, MD - Stanford; Carlene Lugtu, MS-Clinical Informatics Management - STANFORD HEALTH CARE; Zac Eggers, MSN - Stanford Health Care; Ivan Lopez Rodriguez; Kevin Takazawa, n/a - Stanford Health Care; Danyelle Clutter, MBA - Stanford Healthcare; Kyle Vogt, BA - Stanford Health Care; Matthew Rojo, Master of Science - Stanford Medicine; Michael Pfeffer, MD - Stanford University; Christopher Sharp, MD - Stanford University School of Medicine; Jonathan Chen, MD, PhD - Stanford University Hospital; Patricia Garcia, MD - Stanford University, School of Medicine;

---

Automation of Critical Lab Result Communication Improves Lab Efficiency

Presentation Time: 08:45 AM - 09:00 AM

Abstract Keywords: Clinical Process Automation, Innovation in Digital Care, Workflow Efficiency
Primary Track: Industry and Commercial Partnership
Programmatic Theme: Organizational Challenges

An academic health system designed and implemented an automated critical lab results notification system to enhance lab staff efficiency. Based on technologies from Epic, Twilio, UiPath, and Sunquest, a bot called the provider using interactive voice response and, after confirming identity, delivered the results with Epic secure chat. The automation achieved a 27.4% success rate and maintained the 30 minute turnaround time benchmark

Speaker:
Jonathan Austrian, MD
NYU Langone Medical Center

Authors:
Jeffrey Jhang, MD, MBA - NYU Langone Health; Irfan Syed, BS - NYU Langone Health;

---