A new approach to subseasonal multi-model forecasting: Online prediction with expert advice

<div> <p>The benefits of multi-model combinations in climate forecasting have been previously introduced and described for different temporal scales (e.g., Siebert and Stephenson 2019, DelSole 2007, Sansom et al. 2013). Most typical combination methodologies involve weighting strategies that assign each model a constant factor, either uniformly or through a skill assessment. Given that the skill of the models can vary at different timescales, and for multiple reasons (for example, seasonally varying skill, or due to changes in the forecasting system), the fact that these weights remain constant introduces limitations.  </p> </div><div> <p>Within the realm of Machine Learning, a family of algorithms have been developed to perform ‘online prediction with expert advice’ (Cesa-Bianchi et al. 2006). These methods consider a set of weighted ‘experts’ (usually uniformly weighted at the start of the process) to produce subsequent predictions in which the combination or `mixture’ is updated to optimize a loss or skill function.  </p> </div><div> <p>These online forecasting methods potentially have several advantages for their use in climate prediction: </p> </div><div> <div> <ul><li>The fact that the expert combination is updated in every forecast step allows the system to adjust in certain conditions (e.g., the ones mentioned above) to preserve skill; </li> <li>Since a different combination can be easily obtained for different quantiles of the predictand distribution, a robust system can be trained that maximizes skill for its full range. </li> <li>The risk of including incompetent or counterproductive experts is minimized by the fact that the mixture is able to adapt and discard them (or assign them minimal weights). </li> </ul></div> <div> <p>Another potential application of these online prediction methods could be on the design of ‘seamless’ forecasting systems in the sub-seasonal to seasonal sense, which is of interest to several research projects such as S2S4E (https://s2s4e.eu/). For example, the system could be trained with a set of experts that include subsequent launches of a sub-seasonal forecast as well as prior launches of a seasonal forecast. If at any point there is useful information arising from the longer lead time seasonal forecast, the mixture would assign higher weights to it. </p> </div> <div> <p>A set of these online prediction methods have been tested within the S2S4E project and compared to more typical multi-model combination techniques to assess their usefulness for the prediction of country-level energy demand, and potentially other variables. Results show that these innovative methods exhibit significant skill improvements (higher than 5%) with respect to more standard techniques and to individual forecasting systems for lead weeks up to 4.  </p> </div> </div>

Indirect Trust is Simple to Establish

In systems with multiple potentially deceptive agents, any single agent may have to assess the trustworthiness of other agents in order to decide with which agents to interact. In this context, indirect trust refers to trust established through third-party advice. Since the advisers themselves may be deceptive or unreliable, agents need a mechanism to assess and properly incorporate advice. We evaluate existing state-of-the-art methods for computing indirect trust in numerous simulations, demonstrating that the best ones tend to be of prohibitively large complexity. We propose a new and easy to implement method for computing indirect trust, based on a simple prediction with expert advice strategy as is often used in online learning. This method either competes with or outperforms all tested systems in the vast majority of the settings we simulated, while scaling substantially better. Our results demonstrate that existing systems for computing indirect trust are overly complex; the problem can be solved much more efficiently than the literature suggests.