An LLM-Powered Clinical Calculator Chatbot Backed by Verifiable Clinical Calculators and their Metadata

Presentation Time: 08:00 AM - 08:12 AM

Abstract Keywords: Large Language Models (LLMs), Clinical Decision Support, Knowledge Representation and Information Modeling
Primary Track: Foundations
Programmatic Theme: Clinical Informatics

Clinical calculators are widely used, and large language models (LLMs) make it possible to engage them using natural language. We demonstrate a purpose-built chatbot that leverages software implementations of verifiable clinical calculators via LLM tools and metadata about these calculators via retrieval augmented generation (RAG). We compare the chatbot's response accuracy to an unassisted off-the-shelf LLM on four natural language conversation workloads. Our chatbot achieves 100% accuracy on queries interrogating calculator metadata content and shows a significant increase in clinical calculation accuracy vs. the off-the-shelf LLM when prompted with complete sentences (86.4% vs. 61.8%) or with medical shorthand (79.2% vs. 62.0%). It eliminates calculation errors when prompted with complete sentences (0% vs. 16.8%) and greatly reduces them when prompted with medical shorthand (2.4% vs. 18%). While our chatbot is not ready for clinical use, these results show progress in minimizing incorrect calculation results.

Speaker:
Niranjan Kumar, BS
University of Michigan Medical School

Authors:
Niranjan Kumar, BS - University of Michigan Medical School; Farid Seifi, PhD - University of Michigan; Marisa Conte, MLIS - University of Michigan; Allen Flynn, PharmD, PhD - University of Michigan;

---

Measuring accuracy of ConsultBot*, hybrid AI tool, in interpreting blood gas results.

Presentation Time: 08:12 AM - 08:24 AM

Abstract Keywords: Clinical Decision Support, Artificial Intelligence, Evaluation, Large Language Models (LLMs)
Primary Track: Applications
Programmatic Theme: Clinical Informatics

Importance: Arterial and venous blood gases are complex tests used in inpatient settings. While interpretation of blood gas is algorithmic, it is time consuming and error prone when done manually. Our study measures the accuracy of ConsultBot, in automated interpretation of blood gases.
Objective: To determine proportional accuracy of ConsultBot, hybrid AI tool, using rule-based logic combined with large language models (LLMs) in interpreting blood gases.
Design: ConsultBot was tested in IRB-approved single arm trial using a dataset of 101 blood gas results and compared with a predetermined answer-key. The tool's performance was assessed using proportional accuracy, sensitivity, and Cohen's Kappa.
Results: ConsultBot achieved a 98%(99/101)[prop CI 93% - 99.76%] accuracy across the comprehensive database. Sensitivity was 98%[CI 92.9%-99.4%]. Cohen's Kappa of 0.97, suggested high degree of agreement between ConsultBot's interpretation and answer key.
Conclusions: ConsultBot’s evaluation yielded promising results, demonstrating potential in clinical decision support for blood gas interpretations.

Speaker:
Praveen Meka, MD
Dana Farber Cancer Institute

Authors:
Christine Silvers, MD, PhD - Amazon Web Services; Qing Liu, BE - Amazon Web Services; Bharath Gunapati, Sr. Solutions Architect - Amazon Web Services;

---

Identifying Cultural Adaptation Components of a Therapy Chatbot for Spanish-Speaking Family Caregivers

Presentation Time: 08:24 AM - 08:36 AM

Abstract Keywords: Patient Engagement and Preferences, Chronic Care Management, Nursing Informatics, Human-computer Interaction, Personal Health Informatics, Mobile Health, Health Equity
Primary Track: Applications
Programmatic Theme: Consumer Health Informatics

A qualitative study aiming to identify opportunities to culturally adapt COCO (an AI-enhanced application that delivers Problem-Solving Therapy) for Spanish-speaking Latino caregivers of children with chronic conditions. Preliminary findings indicate that cultural adaptation related to health goals and therapy delivery is warranted. The adaptation is feasible and could improve COCO's acceptability, effectiveness, and reach. LLMs can integrate cultural knowledge through methods like RAG, enhancing the chatbot's cultural relevance.

Speaker:
Priscilla Carmiol Rodríguez, MEd, MSN
University of Washington

Authors:
Priscilla Carmiol Rodríguez, MEd, MSN - University of Washington; Genevieve Aguilar, MSN, MPA, BSN, RN - University of Washington; Serena Jinchen Xie, Masters - Biomedical Informatics and Medical Education, University of Washington; Maggie Ramirez, PhD, MS, MS - University of Washington; Weichao Yuwen, PhD, RN - University of Washington Tacoma;

---

Participatory design of a chatbot for sickle cell trait newborn screening results

Presentation Time: 08:36 AM - 08:48 AM

Abstract Keywords: Population Health, User-centered Design Methods, Delivering Health Information and Knowledge to the Public
Primary Track: Applications
Programmatic Theme: Consumer Health Informatics

Newborn screening enables early detection of serious yet treatable conditions. However, parents receiving unexpected carrier results—such as for sickle cell trait—often experience significant distress before speaking with a healthcare provider. To address this wait gap, we developed a scripted chatbot designed to deliver timely, empathetic, and accurate information to support parents and reduce emotional distress. We present initial qualitative and quantitative findings that demonstrate the chatbot’s positive impact as a scalable informatics intervention in the newborn screening workflow.

Speaker:
Karen Eilbeck, PhD
University of Utah

Authors:
Anne Madeo, MS - University of Utah; Courtney Gauchel, Master's of Biomedical Informatics - University of Utah; Kimberley Kaphingst, PhD - University of Utah; Guilherme Del Fiol, MD, PhD - University of Utah; Karen Eilbeck, PhD - University of Utah;

---

Automated Survey Collection with LLM-based Conversational Agents

Presentation Time: 08:48 AM - 09:00 AM

Abstract Keywords: Surveys and Needs Analysis, Artificial Intelligence, Large Language Models (LLMs), Administrative Systems, Usability
Primary Track: Applications
Programmatic Theme: Public Health Informatics

Traditional phone-based surveys are a key method for collecting biomedical and healthcare data, but they are costly, labor-intensive, and challenging to scale. To overcome these limitations, we propose an end-to-end survey collection framework driven by a conversational Large Language Models. Our framework can reduce interviewer burden, enable seamless scalability of survey collection, and it achieves an impressive 98.2% accuracy in interpreting responses from phone surveys.

Speaker:
Kurmanbek Kaiyrbekov, Ph.D.
National Institutes of Health

Authors:
Nicholas Dobbins, PhD, MLIS - Johns Hopkins University; Sean Mooney, Ph.D. - National Institutes of Health;

---

When Patients Go to “Dr. Google” Before They Go to the Emergency Department

Presentation Time: 09:00 AM - 09:12 AM

Abstract Keywords: Delivering Health Information and Knowledge to the Public, Personal Health Informatics, Clinical Decision Support, Clinical Guidelines
Primary Track: Applications
Programmatic Theme: Consumer Health Informatics

Approximately one-third of adults search the internet for health information before visiting an emergency department (ED), with 75% encountering inaccurate content. This study examines how such searches influence patient care. We conducted an observational study of ED visits over a 12-month period, surveying 214 of 576 patients about pre-ED internet use. Data on demographics, comorbidities, acuity, orders, prescriptions, and dispositions were extracted. Patients who searched were typically younger, healthier, and more educated. Most used a general search engine to ask symptom-related questions. Compared to non-searchers, they were less likely to receive lab tests (RR 0.78, p=0.053), imaging (RR 0.75, p=0.094), medications (RR 0.67, p=0.038), or admission (RR 0.68, p=0.175). They were more likely to leave against medical advice (RR 1.67, p=0.067) and receive opioids (RR 1.56, p=0.151). Findings suggest inaccurate health information may contribute to mismatched expectations and altered care delivery.

Speaker:
Enrique Calleros, Undergraduate Student
University of Texas, El Paso

Authors:
Michael Grasso, MD,PhD - University of Maryland School of Medicine; Alexandra Rogalski, MD - University of Maryland School of Medicine; Naveed Farrukh, MD, MPH - The Ohio State University and Nationwide Childrens; Anantaa Kotal, PhD - University of Texas, El Paso; Enrique Calleros, Undergraduate Student - University of Texas, El Paso;

---