Humans and Large Language Models in Clinical Decision Support: A Study with Medical Calculators

Presentation Time: 03:30 PM - 03:45 PM

Abstract Keywords: Clinical Decision Support, Large Language Models (LLMs), Human-computer Interaction, Information Extraction, Evaluation, Data Mining
Primary Track: Applications
Programmatic Theme: Clinical Informatics

Although large language models (LLMs) have been assessed for general medical knowledge using licensing exams, their ability to support clinical decision-making, such as selecting medical calculators, remains uncertain. We assessed nine LLMs, including open-source, proprietary, and domain-specific models, with 1,009 multiple-choice question-answer pairs across 35 clinical calculators and compared LLMs to humans on a subset of questions. While the highest-performing LLM, OpenAI’s o1, provided an answer accuracy of 66.0% (CI: 56.7-75.3%) on the subset of 100 questions, two human annotators nominally outperformed LLMs with an average answer accuracy of 79.5% (CI: 73.5-85.0%). Ultimately, we evaluated medical trainees and LLMs in recommending medical calculators across clinical scenarios like risk stratification and diagnosis. With error analysis showing that the highest-performing LLMs continue to make mistakes in comprehension (49.3% of errors) and calculator knowledge (7.1% of errors), our findings highlight that LLMs are not superior to humans in calculator recommendation.

Authors:
Nicholas Wan, Bachelor of Engineering - National Institutes of Health; Qiao Jin, M.D. - National Institutes of Health; Joey Chan, M.S. - National Library of Medicine; Guangzhi Xiong, BA - University of Virginia; Serina Applebaum, BS - Yale School of Medicine; Aidan Gilson, MD - Massachusetts Eye and Ear; Reid McMurry, MD - Boston Medical Center; Richard Taylor, MD - University of Virginia School of Medicine, Department of Emergency Medicine; Aidong Zhang, PhD - University of Virginia; Qingyu Chen, PhD - Yale University; Zhiyong Lu, PhD - National Library of Medicine, NIH;

---

Technology and Human Support Systems in Decentralized Clinical Trials: A Participant-Centered Case Study in Cystic Fibrosis

Presentation Time: 03:45 PM - 04:00 PM

Abstract Keywords: Human-computer Interaction, Mobile Health, Patient Engagement and Preferences, Qualitative Methods, User-centered Design Methods
Primary Track: Foundations
Programmatic Theme: Clinical Research Informatics

As the integration of informatics into clinical research reshapes the landscape of decentralized clinical trials (DCTs), optimizing participant experience remains a key challenge. Although prior research has established foundations for DCT design, a more comprehensive understanding of participant perspectives is essential to ensure remote methods for data collection meet participant needs. This study contributes to a growing literature in participant-centered DCTs through an analysis of OUTREACH, a 3-month home spirometry study among individuals with cystic fibrosis. Through a qualitative analysis of 46 participant exit interviews, we identified seven themes about participant engagement in OUTREACH, which we organized into three primary categories: motivators, technological infrastructure, and human support. Our findings emphasize the value of reliable technology and comprehensive interpersonal support systems. These findings shed light upon the importance of sociotechnical elements for optimizing participant experience, which may enhance the quality of clinical trial data through meaningful participant engagement.

Authors:
Ayana Sarrieddine, M.S. - University of Washington; Claire Lai, B.S. - Univeristy of Washington; Oliver Bear Don't Walk, PhD - University of Washington; Nick Reid, MHI - Biomedical Informatics and Medical Education, University of Washington; Gregory Sawicki, MD, MPH - Boston Children’s Hospital; Harvard Medical School; Ariel Berlinski, MD - Arkansas Children’s Hospital; Margaret Rosenfeld, MD, MPH - Seattle Children's Hospital; University of Washington; Andrea Hartzler, PhD - University of Washington;

---

My Kidney T.R.E.K. - Thinking, Reflecting, and Empowering Kidney Transplant Patients, through technology.

Presentation Time: 04:00 PM - 04:15 PM

Abstract Keywords: Human-computer Interaction, Patient Engagement and Preferences, Personal Health Informatics, Pediatrics
Primary Track: Applications
Programmatic Theme: Consumer Health Informatics

Adolescents and young adults with kidney transplants face unique challenges as they transition toward independent self-management. These youth need tools that not only support skill-building but also foster reflection, a key component of self-management. To address this need, we created a digital prototype designed to help users reflect on their transplant journey through storytelling and evaluated it in a study with 23 participants (13 youth and 10 caregivers) using. Participants found value in engaging with the prototype, which helped them reflect on their past experiences, gain insight into their current journey, and envision their future. Youth, in particular, reported increased self-awareness and confidence in managing their health. Based on these findings, we present design recommendations for future digital health tools aimed at supporting self-management in youth with chronic conditions.

Authors:
Julia Dunbar, PhD - University of Washington; Wanda Pratt, PhD, FACMI - University of Washington; Lily Jeffs, BS - University of Washington; Sanaa Sayed, n/a - Unversity of Washington; Chelsea Ng, BS - Seattle Children's Hospital; Jodi Smith, MD - Seattle Children's Hospital & University of Washington; Ari Pollack, MD - Seattle Children's Hosp & University of Washington;

---

Automating Patient Safety Workflows: The Development and Implementation of LLaMPS, a Secure Large Language Model Application

Presentation Time: 04:15 PM - 04:30 PM

Abstract Keywords: Large Language Models (LLMs), Natural Language Processing, Patient Safety, Artificial Intelligence, Healthcare Quality
Primary Track: Applications

Despite significant advancements in Generative Artificial Intelligence (GenAI), practical adoption in healthcare, particularly patient safety, remains challenging due to concerns regarding data privacy, model transparency, clinical relevance and user engagement. We present LLaMPS (Large Language Model for Patient Safety), a locally deployed GenAI platform designed to enhance patient safety event management and reporting. LLaMPS integrates automated incident classification, harm-level prediction, intelligent search, and an interactive chatbot. The system employs a Retrieval-Augmented Generation (RAG) approach, leveraging secure, institutionally hosted large language models (LLMs) and a vector database to ensure data privacy and regulatory compliance. Developed iteratively with direct input from clinicians and patient safety experts, LLaMPS demonstrates high classification accuracy and improved user satisfaction, underscoring the potential of locally controlled AI solutions to enhance patient safety workflows.

Authors:
Margaret Zhou, M.S. in Machine Learning and Data Science - Mayo Clinic; Shiba Kuanar, PhD - Mayo Clinic - Rochester; Shrinath Patel, MS - Mayo Clinic; Jennifer Lamers, M.B.A,PMP - Mayo Clinic; Mohsin Ammas, M.B.B.S - Mayo Clinic; Subashnie Devkaran, Ph.D - Mayo Clinic; Joe Nienow, M.B.A - Mayo Clinic; Jill Nagel, PA-C, M.B.A - Mayo Clinic; Kannan Ramar, M.B.B.S.,M.D. - Mayo Clinic; Moein Enayati, Ph.D. - Mayo Clinic; Sean Dowdy, M.D - Mayo Clinic; Che Ngufor, PhD - Mayo Clinic;

---

Telemedicine Access for Young Adults with Recently Diagnosed Type 2 Diabetes

Presentation Time: 04:30 PM - 04:45 PM

Abstract Keywords: Telemedicine, Healthcare Quality, Health Equity
Primary Track: Applications
Programmatic Theme: Consumer Health Informatics

In a study examining newly diagnosed patients with type 2 diabetes between ages 21-45, we found a third of young adults accessed primary care exclusively through telemedicine, most patients that did not seek primary care before their diagnosis used telemedicine in the year after receiving their diagnosis, and modality use was linked to stated patient preferences, food insecurity, and life chaos. We also found that telemedicine-only primary care was associated with higher HbA1c.

Authors:
Christine Board, MPH - Kaiser Permanente Northern California; Jie Huang, Ph.D; Mary Reed, DrPH - Kaiser Permanente Division of Research; Anjali Gopalan, MD - Kaiser Permanente Northern California Division of Research;

---