Who Watches the AI? Evaluating the Performance of an LLM “Jury” for Scalable Monitoring of Deployed AI Systems

Presentation Type: Podium Abstract
Presentation Time: 02:00 PM - 02:12 PM

Abstract Keywords: Artificial Intelligence, Large Language Models (LLMs), Patient Safety, Informatics Implementation
Programmatic Theme: Clinical Informatics

We assigned a four-agent large language model (LLM) jury to perform a fully automated modified Delphi panel, monitoring LLM-generated discharge summaries for errors. The LLM jury achieved consensus in identifying each of 114 errors, including 38 not previously identified by human reviewers. Jury precision, recall, and F1 scores were 0.50, 0.80, and 0.61, respectively. These pilot findings suggest emerging opportunities for LLM juries to screen clinically deployed AI in a manner more scalable than humans.

Speaker(s):
Parnaz Daneshpajouhnejad, MD
University of California, San Francisco
Author(s):
Rayna Kumar, Student - University of California, San Francisco; Benjamin Rosner, MD, PhD, FAMIA - UCSF;

---

Monitoring Models With vs. Without Health-Related Social Needs for Predicting Missed Primary Care Appointments

Presentation Type: Paper - Student
2026 Symposium Health Equity Presentation

Presentation Time: 02:12 PM - 02:24 PM

Abstract Keywords: Informatics Implementation, Health Equity, Fairness and elimination of bias
Programmatic Theme: Clinical Informatics

Healthcare organizations’ access to increasing amounts of health-related social needs (HRSNs) data is an opportunity to enhance predictive analytics in support of effective and efficient care. Missed appointments (i.e., “no-shows”) are detrimental to both patients and healthcare organizations. Identifying patients at high risk of no-shows could lead to more effective care and resource utilization. While HRSNs, including transportation barriers, lead to increased no-show rates, these factors have historically been absent from no-show prediction models. We developed and validated no-show prediction models with and without HRSNs. Models with all features (AUROC: 0.725-0.762; AUPRC: 0.378-0.44) and no HRSNs (AUROC: 0.717-0.753; AUPRC: 0.372-0.431) performed better than those with only HRSNs (AUROC: 0.598-0.652; AUPRC: 0.232-0.306). Percentage of prior no-shows was the most important feature. Over time, model performance and fairness decreased. Future research should test methods for monitoring model performance, develop strategies for implementing models into workflows, and design interventions to assist high-risk individuals.

Speaker(s):
Leigh Anne Tang, PhD
Indiana University/Regenstrief Institute
Author(s):
Leigh Anne Tang, PhD - Indiana University/Regenstrief Institute; Jennylee Swallow, PhD - Indiana University/ Regenstrief Institute; Joshua Vest, PhD - Indiana University;

---

Operationalizing AI Hazards for Learning Health Systems: An Open Hazard-to-Scenario Evidence Library for Healthcare AI Evaluation

Presentation Type: Podium Abstract
Presentation Time: 02:24 PM - 02:36 PM

Abstract Keywords: Artificial Intelligence, Evaluation, Governance, Informatics Implementation
Programmatic Theme: Clinical Informatics

Existing AI governance frameworks and risk taxonomies identify important hazards in healthcare AI but remain largely descriptive and insufficient for operational evaluation. We developed an open hazard-to-scenario evidence library that structures each hazard as a reusable evidence object linking potential harms, safety-critical tasks, candidate measures, mitigation strategies, and simulated-use scenarios. Across 134 hazards in 10 domains, the library provides traceable, simulation-ready informatics infrastructure for reproducible, risk-tiered evaluation in learning health systems.

Speaker(s):
Shauna Overgaard, Ph.D.
Mayo Clinic
Author(s):
Tracey Brereton, Masters - Mayo Clinic;

---

Automated Evaluation can Distinguish the Good and Bad AI Responses to Patient Questions about Hospitalization

Presentation Type: Podium Abstract

Click to View Presentation

2026 Symposium 25x5 Presentation

Presentation Time: 02:36 PM - 02:48 PM

Abstract Keywords: Artificial Intelligence, Evaluation, Natural Language Processing, Delivering Health Information and Knowledge to the Public, Documentation Burden, Large Language Models (LLMs)
Programmatic Theme: Clinical Research Informatics

Automated evaluation of free-text AI answers to patient hospitalization questions could replace slow expert review. Using 100 ArchEHR-QA cases and 2,800 answers from 28 clinical evidence-grounded AI systems, we compared automated factuality and relevance metrics with clinician-informed human ratings across three dimensions (answering the question, using EHR evidence, and using medical knowledge). Metric-derived system rankings closely matched human rankings, enabling scalable benchmarking of patient-centered clinical question answering systems.

Speaker(s):
Sarvesh Soni, PhD
National Library of Medicine (NLM)
Author(s):
Sarvesh Soni, PhD - National Library of Medicine (NLM); Dina Demner-Fushman, MD - National Library of Medicine;

---

To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering

Presentation Type: Paper - Student

Click to View Presentation

Presentation Time: 02:48 PM - 03:00 PM

Abstract Keywords: Artificial Intelligence, Natural Language Processing, Large Language Models (LLMs)
Programmatic Theme: Clinical Research Informatics

To improve the efficiency of medical question answering (MedQA) with large language models (LLMs) by avoiding unnecessary reasoning while maintaining accuracy, we propose Selective Chain-of-Thought (Selective CoT), an inference-time strategy that first predicts whether a question requires reasoning and generates a rationale only when needed. Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated on four biomedical QA benchmarks—HeadQA, MedQA-USMLE, MedMCQA, and PubMedQA. Metrics included accuracy, total generated tokens, and inference time. Selective CoT reduced inference time by 13-45% and token usage by 8-47% with minimal accuracy loss (≤4%). In some model-task pairs, it achieved both higher accuracy and greater efficiency than standard CoT. Compared with fixed-length CoT, Selective CoT reached similar or superior accuracy at substantially lower computational cost. Selective CoT dynamically balances reasoning depth and efficiency by invoking explicit reasoning only when beneficial, reducing redundancy on recall-type questions while preserving interpretability.

Speaker(s):
Zaifu Zhan, MS
University of Minnesota twin cities
Author(s):
Zaifu Zhan, MS - University of Minnesota twin cities; Min Zeng, PhD - University of Minnesota; Shuang Zhou, PhD - University of Minnesota Twin Cities; Yiran Song, doctor - University of Minnesota; Xiaoyi Chen, PhD - University of Minnesota, Twin Cities, MN, USA | Imagine Institute of Genetic Diseases, Paris, France; Yu Hou, PhD - University of Minnesota; Yifan Wu, MS - University of Minnesota; Yang Ruan, MS - University of Minnesota; Rui Zhang, PhD, FACMI, FAMIA, FIAHSI - University of Minnesota, Twin Cities;

---