Patient Perspectives About Deployment of AI Decision Support Tools in a Safety-Net Healthcare System

Presentation Time: 03:30 PM - 03:42 PM

Abstract Keywords: Artificial Intelligence, Clinical Decision Support, Health Equity, Diversity, Equity, Inclusion, and Accessibility, Policy, Qualitative Methods, Surveys and Needs Analysis, Real-World Evidence Generation
Primary Track: Foundations
Programmatic Theme: Clinical Informatics

Despite increasing use of artificial intelligence (AI) in clinical medicine, little is known about patients’ perspectives, particularly in the safety net setting. We surveyed 313 patients on AI awareness, trust, and perceived benefits/risks. Higher AI awareness correlated with greater perceived benefits of clinical AI. Transparency and clinician oversight were key themes underlying trust in clinical AI. Patient perspectives should inform future AI deployment strategies, policy decisions, and governance structures.

Authors:
Avni Kothari, MS - UCSF; Patrick Vossler, PhD - UCSF; Jean Feng, PhD - UCSF; Katherine Steineman, BA - UCSF; Paige Nong, PhD - University of Minnesota; Julia Adler-Milstein, PhD, FACMI - UCSF School of Medicine; Arturo Gasga, MD - UCSF; Seth Goldman, MD - ZSFG/UCSF; Susan Ehrlich, MD - ZSFG / UCSF; Hemal Kanzaria, MD, MSc - UCSF / ZSFG; Lucas Zier, MD, MS - Zuckerberg San Francisco General Hospital;

---

Introducing SPECTACULAR: An Efficient Quantitative Method to Identify Optimal Clinical Decision Support Tool Designs

Presentation Time: 03:42 PM - 03:54 PM

Abstract Keywords: Usability, Clinical Decision Support, Human-computer Interaction, User-centered Design Methods
Primary Track: Applications
Programmatic Theme: Clinical Informatics

This study serves as the 1st implementation of our SPECTACULAR method, which we believe advances the field of human-computer interaction by facilitating efficiency, finding an optimal design, and scaling customization to any setting, condition, or user. Although factorial research designs have been used for technology design, to our knowledge, no one has applied an Adaptive Platform Trial framework to ineffective feature elimination to expedite the design process and potentially reduce the necessary sample size.

Authors:
Bradford Patton, MS - Meharry Medical College; Dagmawi Negesse, M.S in Technology Management with Cyber System Tech Security - Vanderbilt University Medical Center; Satkar Dhakal, MS - Vanderbilt University; Adam Wright, PhD - Vanderbilt University Medical Center; Alvin Jeffery, PhD, RN - Vanderbilt University Medical Center;

---

Leveraging Free-Text Patient-Reported Quality of Life Data for Precision Medicine: Integrating Large Language Model with Mobile Health Data to Support Clinical Decision-Making for Personalized Post-Operative Analgesia

Presentation Time: 03:54 PM - 04:06 PM

Abstract Keywords: Precision Medicine, Patient / Person Generated Health Data (Patient Reported Outcomes), Information Extraction
Primary Track: Applications
Programmatic Theme: Clinical Informatics

Integrating Large Language Models (LLMs) with Patient-Reported Outcomes (PROs) collection systems facilitates a more holistic and integrated understanding of patient health. We integrated an LLM with a mobile data collection system for the structured interpretation of free-text patient-reported quality of life data, aiming to support the creation of a 'digital twin' to advance personalized medicine and digital epidemiology. This integration demonstrated at least 97.4% accuracy in interpreting free-text patient-reported data.

Authors:
Cinzia Anna Maria Papappicco, I have 2 Degrees: 1) Science in Nursing; 2) Computer Science - Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic and Vascular Sciences, University of Padua, Padua, Italy; Corrado Lanera, Mathematics - Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Padova, Italy; Giulia Lorenzoni, Epidemiology - Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Padova, Italy; Nicola Disma, Medicine and Surgery - Unit for Research in Anaesthesia, IRCCS Istituto Giannina Gaslini, Genova, Italy; Dario Gregori, Biostatistics - Unit of Biostatistics, Epidemiology and Public Health, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padova, Padova, Italy;

---

Using Large Language Model to Summarize User Comments to Optimize Clinical Decision Support

Presentation Time: 04:06 PM - 04:18 PM

Abstract Keywords: Clinical Decision Support, Artificial Intelligence, Large Language Models (LLMs)
Primary Track: Applications

Objective: This study evaluated GPT-4's capability to summarize CDS alert comments and improve CDS alerts. Methods: We extracted user comments from alerts at Vanderbilt University Medical Center, generated summaries for 8 alerts by both physicians and GPT-4, and surveyed five CDS experts on clarity, completeness, accuracy, and usefulness. Results: AI-generated summaries matched human summaries in clarity, accuracy, and usefulness while showing significantly higher completeness. Conclusion: GPT-4 effectively distills user feedback to optimize CDS alerts.

Authors:
Allison McCoy, PhD, ACHIP, FACMI, FAMIA - Vanderbilt University Medical Center; Aileen Wright, MD - Vanderbilt; Scott Nelson, PharmD, MS, FAMIA, ACHIP - Vanderbilt University Medical Center; Sean Huang, MD - Vanderbilt University; Hasan Ahmad, DO, MBA, MSc, FACP - Parkview Health; Sabrina Carro, MD - Vanderbilt University Medical Center; Jacob Franklin, M.D. - VUMC; James Brogan, MD, MSc - Vanderbilt University Medical Center; Adam Wright, PhD - Vanderbilt University Medical Center;

---

Speed Matters: Evaluating CDS Hooks Response Times for Pediatric Asthma

Presentation Time: 04:18 PM - 04:30 PM

Abstract Keywords: Clinical Decision Support, Interoperability and Health Information Exchange, Standards
Primary Track: Applications
Programmatic Theme: Clinical Informatics

Speed has long been recognized as an important factor in the success of health information technology (HIT) systems. However, as healthcare moves toward distributed forms of HIT, such as SMART and CDS Hooks, maintaining sub-second response times has become more challenging, particularly for applications that require large amounts of data. In this project we assessed the performance of a CDS Hooks service to support treating patients with asthma in a pediatric primary care setting.

Authors:
James Levens, BS - Children's Hospital of Philadelphia; Chen Kenyon, MD - Children's Hospital of Philadelphia; Jordan Wood, MPH - Children's Hospital of Philadelphia; Lauren Coogle, MD - Children's Hospital of Philadelphia; Dean Karavite, MSI - Children's Hospital of Philadelphia; Laura Aisenberg, MD - Children's Hospital of Philadelphia; Chandler Floyd, BS - Children's Hospital of Philadelphia; Robert Grundmeier, MD - Children's Hospital of Philadelphia;

---

What Isn’t Measured, Can’t be Improved: A Framework for Evaluating CDS Program Effectiveness

Presentation Time: 04:30 PM - 04:42 PM

Abstract Keywords: Clinical Decision Support, Healthcare Quality, Usability
Primary Track: Applications
Programmatic Theme: Clinical Informatics

Clinical Decision Support (CDS) systems are integral to advancing healthcare outcomes by incorporating clinical knowledge and patient data into medical decision-making. Despite their potential, many CDS implementations fail to yield significant improvements, signaling a gap between theoretical benefits and real-world outcomes. We outline the development and application of a novel, comprehensive framework to evaluate the effectiveness of CDS programs within a healthcare institution. The framework, informed by the Donabedian model of healthcare quality, was constructed through a systematic review of internal CDS projects and consultations with a multi-institutional pediatric CDS collaborative. It incorporates structural, process, and outcome metrics, including the number of CDS requests, usability testing rates, and improvements in clinical processes and outcomes. Over three years, 151 CDS requests were analyzed, with 68% implemented. Usability testing increased over the study period, correlating with improved clinical processes and outcomes. The findings underscore the importance of usability testing in enhancing CDS effectiveness and suggest that the framework can serve as a valuable tool for aligning CDS programs with organizational goals and securing necessary resource investments. This structured evaluation approach provides critical insights into optimizing CDS program impacts on healthcare processes and outcomes.

Authors:
Swaminathan Kandaswamy, PhD - Emory University School of Medicine; Sarah Thompson, MSHIMI, BSN, RN - Children's Healthcare of Atlanta; Tonya Bennett, BSHI; Thomas Dawson, PharmD - Children's Healthcare of Atlanta; Evan Orenstein, MD - Children's Healthcare of Atlanta;

---