Improving Alignment Between an Infant Sepsis Prediction Model and User Expectations Using Human-Centered Design Methods

Presentation Time: 02:45 PM - 03:00 PM

Abstract Keywords: Artificial Intelligence/Machine Learning, Human Factors Testing, Usability and Measuring User Experience, Usability and Measuring User Experience
Primary Track: AI and Care Outcomes
Programmatic Theme: Usability, Efficiency, and User Experience

Background: We developed a machine learning model using electronic health record data to improve neonatal sepsis recognition. In developing system designs to study the presentation of model output to clinicians, we applied two human-centered design methods: clinician interviews and rapid prototyping.

Methods: The dynamic rapid prototypes, using nearly 60 patient and model data elements per hour over 72 hours, allowed us to visualize patient data, model predictions, and feature importance. Visualization of this data revealed anomalies such as shifts in model output and discrepancies between feature importance and results from the clinician interviews.

A multidisciplinary team with expertise in neonatology, data science, and human-computer interaction reviewed these anomalies using clinician interview analysis, patient chart reviews, electronic health record data analysis, and model code reviews.

Results: The review process resulted in identifying three categories of anomalies: feature selection, feature importance, and model stability. This process resulted in over 40 changes to the model.

Conclusion: While discovered ad hoc, our experience suggests more rigorous strategies for applying human-centered design methods beyond the presentation of machine learning model output to the development and testing of models.

Speaker:
Alex Ruan, MD
Children's Hospital of Philadelphia

Authors:
Dean Karavite, MSI - Children's Hospital of Philadelphia; Lusha Cao - Children's Hospital of Philadelphia; Mary Catherine Harris, MD - Children's Hospital of Philadelphia; Alex Fidel, MSI - Children's Hospital of Philadelphia; Lyle Ungar - University of Pennsylvania; Gerald Shaeffer - Childrenís Hospital of Philadelphia; Rui Xiao, PhD - Children's Hospital of Philadelphia; Alex Ruan, MD - Children's Hospital of Philadelphia; Patrick Brady, MD - Cincinnati Children's Hospital Medical Center; Heather Kaplan, MD - Cincinnati Children's Hospital Medical Center; Robert Grundmeier, MD - Children's Hospital of Philadelphia;

---

Formative Usability Testing of Designs to Present Machine Learning Output for Improving Sepsis Recognition in Critical Infants

Presentation Time: 03:00 PM - 03:15 PM

Abstract Keywords: Artificial Intelligence/Machine Learning, Human Factors Testing, Usability and Measuring User Experience, Data Visualization
Primary Track: AI and Care Outcomes
Programmatic Theme: Usability, Efficiency, and User Experience

Background: Traditional usability testing evaluates system usability but does not address explainability, which is crucial for systems using machine learning or other forms of artificial intelligence (AI). We developed a machine learning model to improve sepsis recognition in the neonatal intensive care unit (NICU).

Methods: In an ongoing study to explore model representation and use, we developed system mockups using patient and model data from four patients. The mockups utilized nearly 200 data elements per patient and were tested iteratively in a format designed to observe user-system problems, assessment of usability using the Post-Study Scenario System Usability Scale (PSSUQ), and an assessment of explainability informed by published methods extending the Technology Assessment Model (TAM) with new constructs such as trust and understandability.

Previous work in interviewing 30 NICU clinicians identified NICU nurses and advance practice providers (APP) as potentially benefiting most from the model. Thirty clinicians (15 nurses, 15 APP) from two tier four NICUs participated in the test.

Results: Formative testing resulted in seven iterative versions of the system. Testing revealed and addressed usability problems with format, layout, labeling, and support content. Explainability problems identified and addressed include data science terminology, model feature importance, and presenting model data over 24 hours. PSSUQ scores were positive and consistent across all seven versions and overall responses to the TAM based questionnaire indicated high agreement with the explainability of the system.

Conclusion: This study demonstrates that adapting usability testing to include explainability effectively identifies and resolves issues in AI-based systems.

Speaker:
Alex Ruan, MD
Children's Hospital of Philadelphia

Authors:
Dean Karavite, MSI - Children's Hospital of Philadelphia; Alex Fidel, MSI - Children's Hospital of Philadelphia; Mary Catherine Harris, MD - Children's Hospital of Philadelphia; Lusha Cao - Children's Hospital of Philadelphia; Lyle Ungar - University of Pennsylvania; Gerald Shaeffer - Childrenís Hospital of Philadelphia; Rui Xiao, PhD - Children's Hospital of Philadelphia; Alex Ruan, MD - Children's Hospital of Philadelphia; Patrick Brady, MD - Cincinnati Children's Hospital Medical Center; Heather Kaplan, MD - Cincinnati Children's Hospital Medical Center; Robert Grundmeier, MD - Children's Hospital of Philadelphia;

---

Usability of integrated care pathways at a freestanding children’s hospital

2025 Clinical Informatics Conference 25x5 Presentation

Presentation Time: 03:15 PM - 03:30 PM

Abstract Keywords: Usability and Measuring User Experience, Adaptive Clinical Decision Support, EHR Implementation and Optimization
Primary Track: Documentation Burden, Clinician Well-Being and Care Outcomes
Programmatic Theme: Usability, Efficiency, and User Experience

Integrative care pathways (ICPs) are evidence-based, structured care plans used to improve the quality of care and patient outcomes in individuals presenting with a specified clinical problem. While ICPs can potentially be valuable, the healthcare team's usability has historically been a barrier due to the lack of integration into their workflows. AgileMD is an ICP application that integrates into the electronic health record (EHR) and can be utilized to create ICPs for various conditions. In June 2023, Children’s Nebraska, a freestanding children’s hospital, began rolling out ICPs using AgileMD. However, the tool's usability was not assessed among nurses. Therefore, this project aims to describe the usability of the Junctional Ectopic Tachycardia (JET) pathway and the Chylothorax pathway by comparing outcome metrics between the AgileMD pathway and the standard workflow for nurses within the CCU.
The Clinical Effectiveness (CE) team developed a nursing task-driven simulation case for the Chylothorax pathway and the JET pathway. Two test patients were developed in the EHR playground. After completing both case simulations, each nurse completed the NASA-TLX and the System Usability Scale (SUS). A total of 16 CCU nurses completed the study. Results of the NASA-TLX demonstrated a significantly lower cognitive load across all domains for the AgileMD workflow compared to the standard workflow. The SUS score of 91.85 corresponds to an A+ letter grade, indicating “Best imaginable” usability. This study will expand to assess the task completion, click burden, and eye fixation between the two modalities.

Speaker:
Kelsey Zindel, DNP, APRN-NP, CPNP-AC/PC
Children's Nebraska

Authors:
Kelsey Zindel, DNP, APRN-NP, CPNP-AC/PC - Children's Nebraska; Taelyr Weekly, PhD, MPH, BSN, RN - Children's Nebraska; Chris Maloney, MD PhD - Children's Hospital and Medical Center; Ellen Kerns, PhD, MPH - UNMC;

---

Patterns in Viewing of Pediatric Portal Notes

2025 Annual Symposium DEI/Health Equity Presentation

Presentation Time: 03:30 PM - 03:45 PM

Abstract Keywords: 21st Century Cures (including considerations for special populations such as adolescents), Driving Digital Equity, Ethical, Legal, and Social Issues
Primary Track: Innovation in Driving Digital Equity and Social Care
Programmatic Theme: Informatics-Driven Value-Based Healthcare

The 21st Century Cures act mandated the sharing of all clinical notes to patients unless exempted by limited allowable exceptions. However, little is known about who is accessing notes and what types of notes are being viewed. IRB exception was granted for review of metadata from all notes for all patients <25 yo from a multi-state health system between July 2022 – June 2023. We collected information on patient demographics, note types, and author specialties. Continuous variables were summarized using medians and categorical variables as percentages. Further statistical analysis using GEE models pending with significant p <0.05. 1,578,188 unique notes were collected from 419,136 individual patients with 1,269,828 shared on activated patient portals. <1% of notes were blocked by providers based on acceptable exemptions. Notes were more likely to be viewed if patients were younger patients (0-2 years) (24.5%) compared to older (18+) (20.4%), English speaking (24.5%) vs Spanish (14.0%) or Other (19.1%), were privately insured, or considered medically complex. Notes from outpatient specialists were viewed most often (n=156,982, 33%), followed by outpatient primary care (n=103,572, 27%), emergency department (n=11,430, 11%), and inpatient (n=28,923, 9%) notes. Viewership by outpatient note specialty ranged from 3% (Radiology) to 45% (Genetics). Viewership by inpatient note specialty ranged from 3% (Neonatology, Rehabilitation Medicine) to 22% (Anesthesia). The data suggest that demographic and clinical factors influence the viewing of notes. These findings can inform strategies to improve access to information among families of all backgrounds to address the digital divide.

Speaker:
Gift Kopsombut, MD
Nemours

Authors:
Michael Valente, MD - Nemours Children's Health; Samantha Kennedy, MD - Nemours Children's Health; Emily Craver, MS - Mayo Clinic; Sara Slovin, MD - Nemours Children's Health; David West, MD - Nemours Children's Health; Gift Kopsombut, MD - Nemours;

---