Evaluating General Purpose Large Language Models as Autism-Related Behavioral Feature Extractors

Presentation Type: Paper - Student

Click to View Presentation

Presentation Time: 08:00 AM - 08:12 AM

Abstract Keywords: Large Language Models (LLMs), Usability, Clinical Decision Support
Programmatic Theme: Consumer Health Informatics

State-of-the-art remote digital autism assessment systems, including FDA-authorized tools such as Cognoa’s, rely on human-in-the-loop AI frameworks in which annotators rate behavioral features from home videos that are then used by a classifier for diagnosis. However, manual temporal annotation is labor-intensive, costly, and subject to variability. General purpose LLMs can now process video inputs without task-specific training, but their ability to reason over dynamic, temporally evolving clinical behaviors remains unclear. We evaluated two general purpose LLMs on 51 naturalistic videos containing autism-related behaviors, assessing video-level presence detection and temporal localization. Both models achieved moderate-to-high precision (0.71–1.00), but recall was lower (Gemini: 0.58–0.88; Qwen: 0.22–0.47). Temporal recall declined further (Gemini: 46–53%; Qwen: 11–31%), reflecting difficulty capturing intermittent episodes. Common errors included missed subtle behaviors and confusion with normative movements. These findings support integrating LLMs into confidence-aware human-in-the-loop workflows rather than autonomous clinical deployment.

Speaker(s):
Sameer Neupane, PhD
University of California San Francisco
Author(s):
Sameer Neupane, PhD - University of California San Francisco; Mohammadmahdi Honarmand, BS - Department of Mechanical Engineering, Stanford University, Stanford; Dennis Wall, PhD, FACMI - Stanford University, Division of Systems Medicine; Peter Washington, PhD - University of California, San Francisco;

---

Rapid Review of Models Assessing Suicide Risk from Patient Portal and Crisis Text Line Messages

Presentation Type: Paper - Student

Click to View Presentation

Presentation Time: 08:12 AM - 08:24 AM

Abstract Keywords: Patient-/Person-Generated Health Data, Natural Language Processing, Telemedicine, Clinical Decision Support
Programmatic Theme: Public Health Informatics

Despite significant amounts of research on the widespread adoption of patient portals (PPM), research has been limited with respect to how people use PPMs to communicate suicidality. This project looks at previous studies characterizing the communication of suicidality in PPMs and crisis text lines (CTLs). We conducted a rapid scoping review of literature ascertaining suicide risk through PPM or CTL messages. We identified 14 papers describing a total of 87 models. None of the models were implemented and tested for clinical utility. None of the studies used large-language models. Few studies ascertained suicide risk from PPM messages. Future research might assess clinical utility directly or explore large language models in this task. Future modeling efforts might use some of the features and models discovered here to build and implement a new PPM model for suicide risk detection.

Speaker(s):
Hannah Slater, MS
Vanderbilt University Department of Biomedical Informatics
Author(s):
Hannah Slater, MS - Vanderbilt University Department of Biomedical Informatics; Colin Walsh, MD MA - Department of Biomedical Informatics, Vanderbilt University;

---

Predicting Adolescent Suicide Attempts using PHQ-9 Modified for Teens and Structured Electronic Health Records

Presentation Type: Podium Abstract

Click to View Presentation

Presentation Time: 08:24 AM - 08:36 AM

Abstract Keywords: Machine Learning, Clinical Decision Support, Public Health
Programmatic Theme: Public Health Informatics

In a large retrospective cohort of 128,055 adolescents with depression screenings, we tested whether models combining screening data with structured electronic health record (EHR) data can outperform single modality-based models for predicting suicide attempts within 365 days after the screening. A late fusion ensemble combining a screening-based Cox model and an EHR based machine learning model achieved higher discrimination power, precision, and sensitivity, and demonstrated fair performance across demographic subgroups, supporting suicide risk assessment in outpatient care settings.

Speaker(s):
Fuchiang (Rich) Tsui, PhD, FAMIA, IEEE Senior Member
Children's Hospital of Philadelphia and University of Pennsylvania
Author(s):
Eamonn Tweedy, PhD - Children's Hospital of Philadelphia; Victor Ruiz, PhD - Children's Hospital of Philadelphia; Neal D Ryan, MD - University of Pittsburgh; Ran Barzilay, MD, PhD - Children's Hospital of Philadelphia; Jason D Jones, PhD - Children's Hospital of Philadelphia; Molly Davis, PhD - Children’s Hospital of Philadelphia; Nadine Melhem, PhD - University of Pittsburgh; Tami Benton, MD - Children’s Hospital of Philadelphia; Jami Young, PhD - Children’s Hospital of Philadelphia; David Brent, MD - University of Pittsburgh;

---

Identifying Intentional Self-Harm in Longitudinal Electronic Health Record Notes Using Large Language Models

Presentation Type: Podium Abstract

Click to View Presentation

2026 Symposium LIEAF Presentation

Presentation Time: 08:36 AM - 08:48 AM

Abstract Keywords: Large Language Models (LLMs), Quantitative Methods, Artificial Intelligence, Evaluation
Working Group: Academic Forum Community
Programmatic Theme: Academic Informatics / LIEAF

Intentional self-harm is often underreported in structured EHR data. We evaluated 9 large language models (LLMs) to identify self-harm from longitudinal clinical notes for a cohort of 530 patients. While LLMs often exhibited absolute overconfidence in outcome generation, relative confidence rankings effectively stratified prediction reliability. By applying confidence-based thresholds, models achieved significantly better performance. Our findings demonstrate that LLM-generated metrics can reliably optimize clinical outcome validation.

Speaker(s):
Yuxin Wang, MS
Mass General Brigham
Author(s):
Sushama Sreedhara, MSPH - MGB; Mufaddal Mahesri, MPH - MGB; Richard Wyss, Faculty - MGB; Shirley Wang - Harvard Medical School/Brigham & Women's; Kueiyu Joshua Lin, ScD - MGB; Jie Yang, PhD, FACMI, FAMIA - Harvard Medical School;

---

Generative AI for Multi Label Classification: A Suicidality Case Study

Presentation Type: Podium Abstract
Presentation Time: 08:48 AM - 09:00 AM

Abstract Keywords: Machine Learning, Natural Language Processing, Information Extraction
Programmatic Theme: Clinical Research Informatics

Patients frequently present with multiple co-occurring conditions, symptoms, and risk factors that evolve over time, making many clinical phenotyping and prediction tasks inherently multi-label. Here, we investigated the use of multi-label classification (MLC) with generative AI via a case study focused on identifying four suicidality and related factors (SrFs) including suicide ideation (SI), suicide attempts (SA), exposure to suicide (ES), and non-suicidal self-injury (NSSI) from psychiatric electronic health records (EHRs). We developed an end-to-end MLC framework with GPT models and evaluated multiple learning strategies. To address the limitations of conventional evaluation metrics, we introduced label-set-level metrics, along with a multi-label confusion matrix for systematic error analysis. Fine-tuned GPT-3.5 and GPT-4.5 with guided prompting achieved top performance with accuracies of 0.90-0.94, while GPT-4.5 with guided prompting showed superior performance across label sets, including rare or minority label sets, indicating a more balanced and robust performance. Our findings reveal systematic error patterns, such as a tendency toward conservative over-labeling and the conflation of SI and SA. This work demonstrates the feasibility of using generative AI for clinical MLC tasks and provides an alternative solution for identifying information from unstructured EHRs to support large-scale clinical research and evidence-based medicine.

Speaker(s):
Ming Huang, PhD
UTHealth Houston
Author(s):
Ming Huang, PhD - UTHealth Houston; Hua Xu, Ph.D - Yale University; Hongfang Liu, PhD - University of Texas Health Science Center at Houston;

---

A Multi Stakeholder, Iterative Safety Evaluation Framework for a Conversational Agent Responding to Suicidal Ideation

Presentation Type: Podium Abstract

Click to View Presentation

Presentation Time: 09:00 AM - 09:12 AM

Abstract Keywords: Human-computer Interaction, Large Language Models (LLMs), Patient Safety, Evaluation, Patient Engagement and Preferences
Programmatic Theme: Consumer Health Informatics

Conversational agents used in patient care require rigorous safety testing to ensure appropriate responses to emergencies such as suicidal ideation (SI). We developed and implemented a multi‑stakeholder, iterative evaluation framework for a conversational agent deployed in a clinical trial at Kaiser Permanente Southern California supporting patients with mild cognitive impairment or dementia. The protocol uses standardized and clinically authentic SI prompts, an 11‑item rubric, repeated testing, and patient stakeholder review to assess SI detection, response quality, guideline concordance, escalation, and response stability.

Speaker(s):
Claudia Nau, PhD
Kaiser Permanente
Author(s):
Susan Motte, Ms - NewDays Inc; Babak Parviz, PhD - NewDays Inc; Daniel Fenton, PhD - NewDays Inc; Huong Nguyen, RN, PhD - Kaiser Permanente Southern California;

---