scholarly journals Mutual Information as a Performance Measure for Binary Predictors Characterized by Both ROC Curve and PROC Curve Analysis

Entropy ◽  
2020 ◽  
Vol 22 (9) ◽  
pp. 938 ◽  
Author(s):  
Gareth Hughes ◽  
Jennifer Kopetzky ◽  
Neil McRoberts
Keyword(s):  
Mutual Information ◽  
Receiver Operating Characteristic ◽  
Risk Score ◽  
Roc Curve ◽  
Operating Characteristic ◽  
Curve Analysis ◽  
Summary Statistics ◽  
Predictive Values ◽  
Score Threshold ◽  
Receiver Operating

The predictive receiver operating characteristic (PROC) curve differs from the more well-known receiver operating characteristic (ROC) curve in that it provides a basis for the evaluation of binary diagnostic tests using metrics defined conditionally on the outcome of the test rather than metrics defined conditionally on the actual disease status. Application of PROC curve analysis may be hindered by the complex graphical patterns that are sometimes generated. Here we present an information theoretic analysis that allows concurrent evaluation of PROC curves and ROC curves together in a simple graphical format. The analysis is based on the observation that mutual information may be viewed both as a function of ROC curve summary statistics (sensitivity and specificity) and prevalence, and as a function of predictive values and prevalence. Mutual information calculated from a 2 × 2 prediction-realization table for a specified risk score threshold on an ROC curve is the same as the mutual information calculated at the same risk score threshold on a corresponding PROC curve. Thus, for a given value of prevalence, the risk score threshold that maximizes mutual information is the same on both the ROC curve and the corresponding PROC curve. Phytopathologists and clinicians who have previously relied solely on ROC curve summary statistics when formulating risk thresholds for application in practical agricultural or clinical decision-making contexts are thus presented with a methodology that brings predictive values within the scope of that formulation.

scholarly journals On the Binormal Predictive Receiver Operating Characteristic Curve for the Joint Assessment of Positive and Negative Predictive Values

Entropy ◽  
2020 ◽  
Vol 22 (6) ◽  
pp. 593 ◽  
Author(s):  
Gareth Hughes
Keyword(s):  
Receiver Operating Characteristic ◽  
Roc Curve ◽  
Operating Characteristic ◽  
Statistical Evaluation ◽  
Characteristic Curve ◽  
Disease Status ◽  
Curve Analysis ◽  
Roc Curve Analysis ◽  
Predictive Values ◽  
Receiver Operating

The predictive receiver operating characteristic (PROC) curve is a diagrammatic format with application in the statistical evaluation of probabilistic disease forecasts. The PROC curve differs from the more well-known receiver operating characteristic (ROC) curve in that it provides a basis for evaluation using metrics defined conditionally on the outcome of the forecast rather than metrics defined conditionally on the actual disease status. Starting from the binormal ROC curve formulation, an overview of some previously published binormal PROC curves is presented in order to place the PROC curve in the context of other methods used in statistical evaluation of probabilistic disease forecasts based on the analysis of predictive values; in particular, the index of separation (PSEP) and the leaf plot. An information theoretic perspective on evaluation is also outlined. Five straightforward recommendations are made with a view to aiding understanding and interpretation of the sometimes-complex patterns generated by PROC curve analysis. The PROC curve and related analyses augment the perspective provided by traditional ROC curve analysis. Here, the binormal ROC model provides the exemplar for investigation of the PROC curve, but potential application extends to analysis based on other distributional models as well as to empirical analysis.

scholarly journals A Statistical Comparison of the Blossom Blight Forecasts of MARYBLYT and Cougarblight with Receiver Operating Characteristic Curve Analysis

2007 ◽  
Vol 97 (9) ◽  
pp. 1164-1176 ◽  
Author(s):  
M. M. Dewdney ◽  
A. R. Biggs ◽  
W. W. Turechek
Keyword(s):  
Receiver Operating Characteristic ◽  
Roc Curve ◽  
Fire Blight ◽  
Operating Characteristic ◽  
Curve Analysis ◽  
Data Sets ◽  
Risk Threshold ◽  
Geographic Regions ◽  
Pairwise Contrasts ◽  
Receiver Operating

Blossom blight forecasting is an important aspect of fire blight, caused by Erwinia amylovora, management for both apple and pear. A comparison of the forecast accuracy of two common fire blight forecasters, MARYBLYT and Cougarblight, was performed with receiver operating characteristic (ROC) curve analysis and 243 data sets. The rain threshold of Cougarblight was analyzed as a separate model termed Cougarblight and rain. Data were used as a whole and then grouped into geographic regions and cultivar susceptibilities. Frequency distributions of cases and controls, orchards or regions (depending on the data set), with and without observed disease, respectively, in all data sets overlapped. MARYBLYT, Cougarblight, and Cougarblight and rain all predicted blossom blight infection better than chance (P = 0.05). It was found that the blossom blight forecasters performed equivalently in the geographic regions of the east and west coasts of North America and moderately susceptible cultivars based on the 95% confidence intervals and pairwise contrasts of the area under the ROC curve. Significant differences (P < 0.05) between the forecasts of Cougarblight and MARYBLYT were found with pairwise contrasts in the England and very susceptible cultivar data sets. Youden's index was used to determine the optimal cutpoint of both forecasters. The greatest sensitivity and specificity for MARYBLYT coincided with the use of the highest risk threshold for predictions of infection; with Cougarblight, there was no clear single risk threshold across all data sets.

Sign in / Sign up

Export Citation Format

Share Document