Predicting Treatment Attrition in Buprenorphine-Naloxone Therapy: A Machine Learning Approach Using Multi-Site EHR Data

Presentation Time: 08:30 AM - 08:45 AM

Abstract Keywords: Precision Medicine, Machine Learning, Clinical Decision Support
Primary Track: Applications

This study investigates buprenorphine-naloxone (BUP-NAL) treatment attrition by employing machine learning models with multi-site EHR data to predict six-month attrition rates. Comparative analysis between model predictions and clinician predictions underscores the efficacy of machine learning in predicting treatment attrition. The findings advance BUP-NAL treatment strategies, combining data analytics and clinical expertise to detect and assist individuals prone to early treatment discontinuation.

Speaker(s):
Fateme Nateghi Haredasht, PhD
Stanford University

Author(s):

Clinical Utility Profiling (CUP): A practical method for choosing predictive-model cutoff ranges based on target deployment

Presentation Time: 08:45 AM - 09:00 AM

Abstract Keywords: Machine Learning, Clinical Decision Support, Evaluation, Population Health, Diagnostic Systems
Primary Track: Applications
Programmatic Theme: Clinical Research Informatics

Deploying clinical predictive models in practice to support decision-making requires an assessment of fit-for-use, including predictive performance, clinical value, and applicability to the target population. Current evaluation approaches lack semantic alignment in cost tradeoffs and assessment in use. Stemmed from decision theory, Clinical Utility Profiling (CUP) aims to assess a model’s value by outlining potential success and failure modes with regard to preferences, patient utility, and applicable range of disease prevalence in the target population.

Speaker(s):
Star Liu, M.S. in Biomedical Informatics and Data Science
Johns Hopkins School of Medicine - Biomedical Informatics and Data Science

Author(s):
Harold Lehmann, MD, PhD - Johns Hopkins University;

Federated Multiple Imputation for Variables that Are Missing Not At Random in Distributed Electronic Health Records

Presentation Time: 09:00 AM - 09:15 AM

Abstract Keywords: Machine Learning, Privacy and Security, Data Mining
Primary Track: Foundations
Programmatic Theme: Clinical Informatics

Large electronic health records (EHR) have been widely implemented and are available for research activities. The magnitude of such databases often requires storage and computing infrastructure that are distributed at different sites. Restrictions on data-sharing due to privacy concerns have been another driving force behind the development of a large class of distributed and/or federated machine learning methods. While missing data problem is also present in distributed EHRs, albeit potentially more complex, distributed multiple imputation methods have not received as much attention. An important advantage of distributed multiple imputation, as well as distributed analysis, is that it allows researchers to borrow strength across data sites, mitigating potential fairness issues for minority groups that do not have enough volume at certain sites. In this paper, we propose a communication-efficient and privacy-preserving distributed multiple imputation algorithms for variables that are missing not at random.

Speaker(s):
Yi Lian, PhD
University of Pennsylvania

Author(s):
Yi Lian, PhD - University of Pennsylvania; Xiaoqian Jiang, PhD - University of Texas Health Science Center at Houston; Qi Long, Ph.D. - University of Pennsylvania;

“Why did the AUC drop?” A Hierarchical Framework to Explain Performance Changes of Machine Learning Models across Hospital Sites

Presentation Time: 09:15 AM - 09:30 AM

Abstract Keywords: Machine Learning, Evaluation, Clinical Decision Support, Interoperability and Health Information Exchange, Precision Medicine, Governance of Artificial Intelligence
Primary Track: Foundations
Programmatic Theme: Clinical Informatics

Machine learning models often differ in performance across hospitals due to changes, e.g., in patient population or practice patterns. Understanding which factors most explain the performance difference is crucial for deciding how to close the performance gap. We introduce a hierarchical framework that quantifies the contribution of variables to difference in a model’s AUC between hospitals. The framework is then illustrated on a model to predict acute care needs in radiation therapy patients at two hospitals.

Speaker(s):
Harvineet Singh, Ph.D.
University of California, San Francisco

Author(s):
Harvineet Singh, Ph.D. - University of California, San Francisco; Andrew Chuang, BS - University of California, San Francisco; Fan Xia, PhD - University of California, San Francisco; Adarsh Subbaswamy, PhD - US Food and Drug Administration; Alexej Gossmann; Nicholas Petrick; Berkman Sahiner - FDA/CDRH; Gene Pennello, PhD; Mi-Ok Kim, PhD - University of California, San Francisco; Romain Pirracchio, MD, PhD - University of California, San Francisco; Ryzen Benson, PhD - UCSF; Marianna Elia, MSE - University of California, San Francisco; Manisha Palta, MD - Duke University; Julian Hong, M.D., M.S. - UCSF; Jean Feng, PhD;

A Three-Part Framework for Evaluating Clinical Prediction Models

Presentation Time: 09:30 AM - 09:45 AM

Abstract Keywords: Evaluation, Machine Learning, Informatics Implementation, Fairness and Elimination of Bias, Health Equity, Clinical Decision Support, Bioinformatics, Pediatrics
Primary Track: Applications
Programmatic Theme: Clinical Informatics

Despite the proliferation of clinical prediction models, few are implemented in-practice and even fewer lead to improved patient care. We propose a framework which unifies three conceptually independent evaluation methods, and we use a specific model as a case-study to demonstrate how this framework can be applied to comprehensively evaluate the usefulness of a clinical prediction model.

Speaker(s):
Ashley Oliver, MPH


Author(s):
Abdul Tariq, PhD - Children's Hospital of Philadelphia; Hojjat Salmasian, MD, MPH, PhD, FAMIA - Children's Hospital of Philadelphia;

Local Evaluation of Predictive Models at US Hospitals

Presentation Time: 09:45 AM - 10:00 AM

Abstract Keywords: Legal, Ethical, Social and Regulatory Issues, Governance of Artificial Intelligence, Clinical Decision Support
Primary Track: Policy
Programmatic Theme: Clinical Informatics

Local evaluation of predictive models, machine learning, and artificial intelligence (AI) to ensure accuracy and guard against bias is a widely accepted goal in health informatics. We describe the current state of use and governance of predictive models among US hospitals based on national survey data from 2023 (n=2,425). Overall, 65% of hospitals reported using machine learning or predictive models. Among these hospitals, 63% reported that most or all models were evaluated for accuracy in data from their health system (local evaluation) and 46% reported that most or all models were locally evaluated for bias. Rates of local evaluation varied by the sources of models used by the hospital, with hospitals that did not develop their own models reporting much lower rates of evaluation.

Speaker(s):
Jordan Everson
Office of the National Coordinator for Health Information Technology

Author(s):