A Review of Probabilistic Opinion Pooling Algorithms with Application to Insider Threat Detection

We retrospectively explore the effectiveness of various probabilistic opinion pools against a set of insider threat detection modeling data from a recently completed, multiyear, sponsored research effort. We explored four opinion pools: the linear opinion pool (likely the most popular), the beta-transformed linear opinion pool, the geometric opinion pool, and a multiplicative method based on odds called Bordley’s formula. The data for our study came from our recent work in the inference of insider threats for our research sponsor. In this work, we created a multimodeling inference enterprise modeling (MIEM) process to either predict threats within a population or, given the threats, predict how well the enterprise system can detect those threats. As part of larger research challenges designed by the research sponsor, we applied the MIEM process quarterly to respond to a sequence of varying challenge problems (CPs). Via MIEM, we developed multiple, independent computation forecast models. These models generated certainty intervals to answer CP questions. These intervals were fused into a single interval for each question via an expert panel prior to submission. The sponsors scored the responses against ground truth. In this paper, we (a) ask which pooling functions work best on these data and consider why, and (b) compare this performance to the actual submissions to determine if one of the pooling functions performed better than our judgment-based fusion.

Forecast densities for economic aggregates from disaggregate ensembles

We extend the “bottom up” approach for forecasting economic aggregates with disaggregates to probability forecasting. Our methodology utilises a linear opinion pool to combine the forecast densities from many disaggregate forecasting specifications, using weights based on the continuous ranked probability score. We also adopt a post-processing step prior to forecast combination. These methods are adapted from the meteorology literature. In our application, we use our approach to forecast US Personal Consumption Expenditure inflation from 1990q1 to 2009q4. Our ensemble combining the evidence from 16 disaggregate PCE series outperforms an integrated moving average specification for aggregate inflation in terms of density forecasting.

Methods For Combining Experts' Probability Assessments

This article reviews statistical techniques for combining multiple probability distributions. The framework is that of a decision maker who consults several experts regarding some events. The experts express their opinions in the form of probability distributions. The decision maker must aggregate the experts' distributions into a single distribution that can be used for decision making. Two classes of aggregation methods are reviewed. When using a supra Bayesian procedure, the decision maker treats the expert opinions as data that may be combined with its own prior distribution via Bayes' rule. When using a linear opinion pool, the decision maker forms a linear combination of the expert opinions. The major feature that makes the aggregation of expert opinions difficult is the high correlation or dependence that typically occurs among these opinions. A theme of this paper is the need for training procedures that result in experts with relatively independent opinions or for aggregation methods that implicitly or explicitly model the dependence among the experts. Analyses are presented that show that m dependent experts are worth the same as k independent experts where k ≤ m. In some cases, an exact value for k can be given; in other cases, lower and upper bounds can be placed on k.