Early Patient Safety Report Triaging Using Large Language Models

2025 Clinical Informatics Conference On Demand

2025 Clinical Informatics Conference 25x5 Presentation

Presentation Time: 01:00 PM - 01:15 PM

Abstract Keywords: Patient Safety, Artificial Intelligence/Machine Learning, Documentation Burden, Workflow Efficiency
Primary Track: Documentation Burden, Clinician Well-Being and Care Outcomes
Programmatic Theme: Informatics-Driven Value-Based Healthcare

Manual triage of patient safety reports (PSRs) is inefficient and subjective. This study applied large language models (LLMs) to automate event-type and harm-level classification over 71,287 PSRs. Three approaches of sequence text classification, text label generation, and multi-output generation were evaluated, with sequence text classification achieving the highest performance (Accuracy:0.88, F1: 0.83) for event-type prediction. Although harm-level predictions remain challenging, LLMs show significant potential for improving efficiency and standardization in patient safety monitoring.

Speaker:
Moein Enayati, PhD
Mayo Clinic

Authors:
Moein Enayati, PhD - Mayo Clinic; Margaret Zhou, MS - Mayo Clinic; Shrinath Patel, MS - Mayo Clinic; Gavin Schaeferle, MS - Mayo Clinic; Shiba Kuanar, PhD - Mayo Clinic - Rochester; Jennifer Lamers, MBA - Mayo Clinic; Joe Nienow, MS - Mayo Clinic; Kannan Ramar, MD, MBA - Mayo Clinic; Jill Nagel, MBA - Mayo Clinic; Che Ngufor, PhD - Mayo Clinic;

Transformative Role of Generative Artificial Intelligence in Inpatient Medicine: Real-World Implementations and Future Directions

2025 Clinical Informatics Conference On Demand

Presentation Time: 01:15 PM - 01:30 PM

Abstract Keywords: Artificial Intelligence/Machine Learning, Documentation Burden, Workflow Efficiency, Patient Safety, Big Data, Data Science, Clinician Burnout
Primary Track: AI and Care Outcomes
Programmatic Theme: Emerging Technology and Technical Infrastructure

This presentation explores the transformative potential of Generative Artificial Intelligence (GenAI) in inpatient medicine, focusing on real-world implementations at NYU Langone Health. Attendees will learn about successful applications, including automatic hospital course generation, patient-friendly discharge narratives, secure chat classification and using GPT to improve medication safety. The session will also address responsible use, safety measures, and end-user education, aiming to enhance patient care and clinician efficiency.

Speaker:
Paawan Punjabi, MD, MSc
New York University School of Medicine/NYU Langone Health System

Authors:
Paawan Punjabi, MD, MSc - New York University School of Medicine/NYU Langone Health System; William Small, MD, MBA - NYU Langone Health; Jonah Feldman, MD, FACP - NYU Langone Health; Jonah Zaretsky, MD - NYU Langone Hospital Brooklyn; Priyanka Solanki, MD - NYU; Jonathan Austrian, MD - NYU Langone Medical Center; Paul Testa, MD, JD, MPH - NYU Langone Health; Yindalon Aphinyanaphongs, MD - NYU Langone Health;

AI-Driven Automation of Procedural Case Log Documentation

2025 Clinical Informatics Conference On Demand

2025 Clinical Informatics Conference 25x5 Presentation

Presentation Time: 01:30 PM - 01:45 PM

Abstract Keywords: Documentation Burden, Data Science, Big Data, Ambient documentation, Artificial Intelligence/Machine Learning
Primary Track: Documentation Burden, Clinician Well-Being and Care Outcomes
Programmatic Theme: Emerging Technology and Technical Infrastructure

Physician case logs are crucial for certification, credentialing, and monitoring medical training, yet their documentation in radiology remains a labor-intensive, manual process. Residents often record over 6,000 procedures manually during their training, which not only diverts time from clinical activities but also increases the likelihood of errors. This inefficiency hampers both the residents and their training programs, highlighting the need for automated solutions. To address this, we employed large language models (LLMs) to automate case log documentation by processing procedural reports and answering predefined questions across three sections: Vascular Diagnosis, Vascular Intervention, and Nonvascular Intervention. Using Meditron-70B and MedLLaMA2-7B, we evaluated their performance against gold-standard annotations from a trained physician. Meditron-70B demonstrated superior accuracy, achieving an F1-score of 72.21% compared to MedLLaMA2's 30.90%. It excelled in precision (>88%) and recall (>59%), particularly in vascular tasks, underscoring its potential for automating this critical process.

Future work will focus on fine-tuning LLMs using ground truth data to enhance alignment with case log requirements. Additionally, we plan to expand the dataset beyond the initial 60 reports to thousands of cases, enabling more comprehensive evaluations and improvements. By integrating more robust architectures and scaling the analysis, we aim to create a reliable solution that reduces documentation errors, saves time, and improves the overall efficiency of medical training workflows. This approach paves the way for transforming case log documentation into an efficient and error-free process.

Speaker:
Nafiz Imtiaz Khan, Student
University of California - Davis

Authors:
Nafiz Imtiaz Khan, Student - University of California - Davis; Vladimir Filkov, PhD - UC Davis; Roger Eric Goldman, PhD - UC Davis;

Implementing AI-Driven Patient Summarization in Electronic Health Records: Early Insights, Best Practices, and Impact Evaluation

2025 Clinical Informatics Conference On Demand

2025 Clinical Informatics Conference 25x5 Presentation

Presentation Time: 01:45 PM - 02:00 PM

Abstract Keywords: Workflow Efficiency, Clinician Burnout, Artificial Intelligence/Machine Learning, Documentation Burden
Primary Track: Documentation Burden, Clinician Well-Being and Care Outcomes
Programmatic Theme: Usability, Efficiency, and User Experience

The integration of generative artificial intelligence (AI) in healthcare promises to alleviate documentation burdens and enhance patient care. At the Children’s Hospital of Philadelphia (CHOP), we integrated an "AI Note Summarization" tool into our electronic health record to automate the extraction of patient histories from existing notes. This pilot, involving 22 clinical staff in an outpatient setting, leveraged ChatGPT-4o within a HIPAA-compliant framework to generate concise narrative summaries. Since June of 2024, our pilot group has been assisting our AI-working group in assessing the tool's effectiveness, safety, and overall experience. The tool has been live in production environments since August of 2024, with ongoing pilot user testing and feedback.

Our pilot results have been compelling. Over 2,370 summaries were generated from more than 32,426 notes with a minimal cancellation rate of 0.5%. User feedback indicates a high 70% satisfaction rate, with 90% of users affirming enhanced clinical workflows and 50% reporting increased insights into their patient's histories. Continuous bi-monthly reviews by the CHOP AI team help refine the tool, focusing on the tool's reliability and utility in clinical practice.

Speaker:
Osvaldo Mercado, MD
Children's Hospital of Philadelphia

Authors:
Shikha Sinha, MD - Children's Hospital of Philadelphia; Greg Lawton, MD - Children's Hospital of Philadelphia; Stephon Proctor, PhD, ABPP - Childrenís Hospital of Philadelphia;