CMU Artificial Intelligence Seminar Series ^{sponsored by}

Tuesday, March 22, 2022

Time: 12:00 - 01:00 PM ET

Chirag Gupta -- Provably calibrating ML classifiers without distributional assumptions

Relevant Paper(s):

• Top-label calibration and multiclass-to-binary reductions
• Distribution-free calibration guarantees for histogram binning without sample splitting
• Distribution-free binary classification: prediction sets, confidence intervals and calibration

Abstract: Most ML classifiers provide probability scores for the different classes. What do these scores mean? Probabilistic classifiers are said to be calibrated if the observed frequencies of labels match the claimed/reported probabilities. While calibration in the binary classification setting has been studied since the mid-1900s, there is less clarity on the right notion of calibration for multiclass classification. In this talk, I will present recent work where we investigate the relationship between commonly considered notions of multiclass calibration and the calibration algorithms used to achieve these notions. We will discuss our proposed notion of top-label calibration, and the general framework of multiclass-to-binary (M2B) calibration. We show that any M2B notion of calibration can be provably achieved, no matter how the data is distributed. I will present these calibration guarantees as well as experimental results on calibrating deep learning models. Our proposed algorithms beat existing algorithms in most situations. Code for this work is available at https://github.com/aigen/df-posthoc-calibration.