Derivation and Experimental Performance of Standard and Novel Uncertainty Calibration Techniques

Presentation Time: 08:30 AM - 08:45 AM

Abstract Keywords: Machine Learning, Deep Learning, Data Mining, Evaluation
Primary Track: Foundations
Programmatic Theme: Clinical Informatics

To aid in the transparency of state-of-the-art machine learning models, there has been considerable research performed in uncertainty quantification (UQ). UQ aims to quantify what a model does not know by measuring variation of the model under stochastic conditions and has been demonstrated to be a potentially powerful tool for medical AI. Evaluation of UQ, however, is largely constrained to visual analysis. In this work, we expand upon the Rejection Classification Index (RC-Index) and introduce the relative RC-Index as measures of uncertainty based on rejection classification curves. We hypothesize that rejection classification curves can be used as a basis to derive a metric of how well a given arbitrary uncertainty quantification metric can identify potentially incorrect predictions by an ML model. We compare RC-Index and rRC-Index to established measures based on lift curves.

Speaker(s):
Katherine (Katie) Brown, PhD
Vanderbilt University Medical Center

Author(s):
Katherine (Katie) Brown, PhD - Vanderbilt University Medical Center; Steven Talbert, PhD - University of Central Florida; Douglas Talbert, PhD - Tennessee Tech University;