Stable reliability diagrams for probabilistic classifiers

A probability forecast or probabilistic classifier is reliable or calibrated if the predicted probabilities are matched by ex post observed frequencies, as examined visually in reliability diagrams. The classical binning and counting approach to plotting reliability diagrams has been hampered by a lack of stability under unavoidable, ad hoc implementation decisions. Here, we introduce the CORP approach, which generates provably statistically consistent, optimally binned, and reproducible reliability diagrams in an automated way. CORP is based on nonparametric isotonic regression and implemented via the pool-adjacent-violators (PAV) algorithm—essentially, the CORP reliability diagram shows the graph of the PAV-(re)calibrated forecast probabilities. The CORP approach allows for uncertainty quantification via either resampling techniques or asymptotic theory, furnishes a numerical measure of miscalibration, and provides a CORP-based Brier-score decomposition that generalizes to any proper scoring rule. We anticipate that judicious uses of the PAV algorithm yield improved tools for diagnostics and inference for a very wide range of statistical and machine learning methods.

A Note on Sandroni-Shmaya Belief Elicitation Mechanism

Incentive-compatible methods for eliciting beliefs, such as proper scoring rules, often rely on strong assumptions about how humans behave when making decisions under risk and uncertainty. For example, standard proper scoring rules assume that humans are risk neutral, an assumption that is often violated in practice. Under such an assumption, proper scoring rules induce honest reporting of beliefs, in a sense that experts maximize their expected scores from a proper scoring rule by honestly reporting their beliefs.Sandroni and Shmaya [Economic Theory Bulletin, volume 1, issue 1, 2013] suggested a remarkable mechanism based on proper scoring rules that induces honest reporting of beliefs without any assumptions on experts’ risk attitudes. In particular, the authors claimed that the mechanism relies only on the natural assumptions of probabilistic sophistication and dominance. We suggest in this paper that the reduction of compound lotteries axiom is another assumption required for Sandroni and Shmaya’s mechanism to induce honest reporting of beliefs. We further elaborate on the implications of such an extra assumption in light of recent findings regarding the reduction of compound lotteries axiom.