“Why did the AUC drop?” A Hierarchical Framework to Explain Performance Changes of Machine Learning Models across Hospital Sites

Presentation Time: 09:15 AM - 09:30 AM

Abstract Keywords: Machine Learning, Evaluation, Clinical Decision Support, Interoperability and Health Information Exchange, Precision Medicine, Governance of Artificial Intelligence
Primary Track: Foundations
Programmatic Theme: Clinical Informatics

Machine learning models often differ in performance across hospitals due to changes, e.g., in patient population or practice patterns. Understanding which factors most explain the performance difference is crucial for deciding how to close the performance gap. We introduce a hierarchical framework that quantifies the contribution of variables to difference in a model’s AUC between hospitals. The framework is then illustrated on a model to predict acute care needs in radiation therapy patients at two hospitals.

Speaker(s):
Harvineet Singh, Ph.D.
University of California, San Francisco

Author(s):
Harvineet Singh, Ph.D. - University of California, San Francisco; Andrew Chuang, BS - University of California, San Francisco; Fan Xia, PhD - University of California, San Francisco; Adarsh Subbaswamy, PhD - US Food and Drug Administration; Alexej Gossmann; Nicholas Petrick; Berkman Sahiner - FDA/CDRH; Gene Pennello, PhD; Mi-Ok Kim, PhD - University of California, San Francisco; Romain Pirracchio, MD, PhD - University of California, San Francisco; Ryzen Benson, PhD - UCSF; Marianna Elia, MSE - University of California, San Francisco; Manisha Palta, MD - Duke University; Julian Hong, M.D., M.S. - UCSF; Jean Feng, PhD;