Proper scoring rules for interval probabilistic forecasts

In this paper, we develop strictly proper scoring rules that may be used to evaluate the accuracy of a sequence of probabilistic forecasts. In practice, when forecasts are submitted for multiple uncertainties, competing forecasts are ranked by their cumulative or average score. Alternatively, one could score the implied joint distributions. We demonstrate that these measures of forecast accuracy disagree under some commonly used rules. Furthermore, and most importantly, we show that forecast rankings can depend on the selected scoring procedure. In other words, under some scoring rules, the relative ranking of probabilistic forecasts does not depend solely on the information content of those forecasts and the observed outcome. Instead, the relative ranking of forecasts is a function of the process by which those forecasts are evaluated. As an alternative, we describe additive and strongly additive strictly proper scoring rules, which have the property that the score for the joint distribution is equal to a sum of scores for the associated marginal and conditional distributions. We give methods for constructing additive rules and demonstrate that the logarithmic score is the only strongly additive rule. Finally, we connect the additive properties of scoring rules with analogous properties for a general class of entropy measures.

Download Full-text

A construction principle for proper scoring rules

Proceedings of the American Mathematical Society Series B ◽

10.1090/bproc/98 ◽

2021 ◽

Vol 8 (24) ◽

pp. 297-301

Author(s):

Jonas Brehmer

Keyword(s):

Predictive Distribution ◽

Scoring Rules ◽

Parametric Family ◽

Scoring Rule ◽

Proper Scoring Rules ◽

Construction Principle ◽

Probabilistic Forecasts ◽

New Scoring

Proper scoring rules enable decision-theoretically principled comparisons of probabilistic forecasts. New scoring rules can be constructed by identifying the predictive distribution with an element of a parametric family and then applying a known scoring rule. We introduce a condition which ensures propriety in this construction and thereby obtain novel proper scoring rules.

Download Full-text

Beyond Strictly Proper Scoring Rules: The Importance of Being Local

Weather and Forecasting ◽

10.1175/waf-d-19-0205.1 ◽

2021 ◽

Author(s):

Hailiang Du

Keyword(s):

Scoring Rules ◽

Forecast Performance ◽

Locality Property ◽

Proper Scoring Rules ◽

Statistical Measures ◽

Probabilistic Forecasts ◽

Direct Interpretation ◽

Probability Mass ◽

Smooth Transformation

AbstractThe evaluation of probabilistic forecasts plays a central role both in the interpretation and in the use of forecast systems and their development. Probabilistic scores (scoring rules) provide statistical measures to assess the quality of probabilistic forecasts. Often, many probabilistic forecast systems are available while evaluations of their performance are not standardized, with different scoring rules being used to measure different aspects of forecast performance. Even when the discussion is restricted to strictly proper scoring rules, there remains considerable variability between them; indeed strictly proper scoring rules need not rank competing forecast systems in the same order when none of these systems are perfect. The locality property is explored to further distinguish scoring rules. The nonlocal strictly proper scoring rules considered are shown to have a property that can produce “unfortunate” evaluations. Particularly the fact that Continuous Rank Probability Score prefers the outcome close to the median of the forecast distribution regardless the probability mass assigned to the value at/near the median raises concern to its use. The only local strictly proper scoring rules, the logarithmic score, has direct interpretations in terms of probabilities and bits of information. The nonlocal strictly proper scoring rules, on the other hand, lack meaningful direct interpretation for decision support. The logarithmic score is also shown to be invariant under smooth transformation of the forecast variable, while the nonlocal strictly proper scoring rules considered may, however, change their preferences due to the transformation. It is therefore suggested that the logarithmic score always be included in the evaluation of probabilistic forecasts.

Download Full-text

Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities*

Monthly Weather Review ◽

10.1175/mwr-d-14-00269.1 ◽

2015 ◽

Vol 143 (4) ◽

pp. 1321-1334 ◽

Cited By ~ 53

Author(s):

Michael Scheuerer ◽

Thomas M. Hamill

Keyword(s):

Predictive Performance ◽

Predictive Distribution ◽

Scoring Rules ◽

Finite Sample ◽

Ensemble Forecasts ◽

Proper Scoring Rules ◽

Probabilistic Forecasts ◽

Multivariate Generalization ◽

Wind Park

Abstract Proper scoring rules provide a theoretically principled framework for the quantitative assessment of the predictive performance of probabilistic forecasts. While a wide selection of such scoring rules for univariate quantities exists, there are only few scoring rules for multivariate quantities, and many of them require that forecasts are given in the form of a probability density function. The energy score, a multivariate generalization of the continuous ranked probability score, is the only commonly used score that is applicable in the important case of ensemble forecasts, where the multivariate predictive distribution is represented by a finite sample. Unfortunately, its ability to detect incorrectly specified correlations between the components of the multivariate quantity is somewhat limited. In this paper the authors present an alternative class of proper scoring rules based on the geostatistical concept of variograms. The sensitivity of these variogram-based scoring rules to incorrectly predicted means, variances, and correlations is studied in a number of examples with simulated observations and forecasts; they are shown to be distinctly more discriminative with respect to the correlation structure. This conclusion is confirmed in a case study with postprocessed wind speed forecasts at five wind park locations in Colorado.

Download Full-text

Strictly Proper Scoring Rules and the Wisdom of the Crowd

SSRN Electronic Journal ◽

10.2139/ssrn.2261272 ◽

2013 ◽

Author(s):

Fabian Krueger ◽

Peter Schanbacher

Keyword(s):

Scoring Rules ◽

Proper Scoring Rules

Download Full-text

Evaluating probabilistic forecasts of football matches: the case against the ranked probability score

Journal of Quantitative Analysis in Sports ◽

10.1515/jqas-2019-0089 ◽

2021 ◽

Vol 0 (0) ◽

Author(s):

Edward Wheatcroft

Keyword(s):

Scoring Rules ◽

Brier Score ◽

Sporting Events ◽

Forecast Performance ◽

Scoring Rule ◽

Probability Score ◽

Probabilistic Forecasts ◽

Non Local ◽

Evaluating Forecasts ◽

Non Locality

Abstract A scoring rule is a function of a probabilistic forecast and a corresponding outcome used to evaluate forecast performance. There is some debate as to which scoring rules are most appropriate for evaluating forecasts of sporting events. This paper focuses on forecasts of the outcomes of football matches. The ranked probability score (RPS) is often recommended since it is ‘sensitive to distance’, that is it takes into account the ordering in the outcomes (a home win is ‘closer’ to a draw than it is to an away win). In this paper, this reasoning is disputed on the basis that it adds nothing in terms of the usual aims of using scoring rules. A local scoring rule is one that only takes the probability placed on the outcome into consideration. Two simulation experiments are carried out to compare the performance of the RPS, which is non-local and sensitive to distance, the Brier score, which is non-local and insensitive to distance, and the Ignorance score, which is local and insensitive to distance. The Ignorance score outperforms both the RPS and the Brier score, casting doubt on the value of non-locality and sensitivity to distance as properties of scoring rules in this context.

Download Full-text