The Simpson’s Paradox in the Offline Evaluation of Recommendation Systems

Recommendation systems are often evaluated based on user’s interactions that were collected from an existing, already deployed recommendation system. In this situation, users only provide feedback on the exposed items and they may not leave feedback on other items since they have not been exposed to them by the deployed system. As a result, the collected feedback dataset that is used to evaluate a new model is influenced by the deployed system, as a form of closed loop feedback. In this article, we show that the typical offline evaluation of recommender systems suffers from the so-called Simpson’s paradox. Simpson’s paradox is the name given to a phenomenon observed when a significant trend appears in several different sub-populations of observational data but disappears or is even reversed when these sub-populations are combined together. Our in-depth experiments based on stratified sampling reveal that a very small minority of items that are frequently exposed by the deployed system plays a confounding factor in the offline evaluation of recommendation systems. In addition, we propose a novel evaluation methodology that takes into account the confounder, i.e., the deployed system’s characteristics. Using the relative comparison of many recommendation models as in the typical offline evaluation of recommender systems, and based on the Kendall rank correlation coefficient, we show that our proposed evaluation methodology exhibits statistically significant improvements of 14% and 40% on the examined open loop datasets (Yahoo! and Coat), respectively, in reflecting the true ranking of systems with an open loop (randomised) evaluation in comparison to the standard evaluation.

THE PARADOX BETWEEN TRUTH AND LIES

In today’s age of fast and multifarious publications, we often come across such paradoxes, for example, that research and statistical analysis may indeed have been carried out correctly (properly), but the interpretation of the results is inadequate, or even incorrect, or misleading. To relate this hypothesis to Moravec’s paradox, one could reformulate the latter by saying that ‘to do research is easy; to discuss results is difficult’. But why is this so? Let us consider this issue from the perspective of another paradox, Simpson’s paradox.

On Simpson’s paradox: To remain or not to remain a population-based science

In this article, I critically reflect on J. F. Arocha’s (2021) contention that psychologists need to use methods and tools that are suitable for data analysis at the individual level. First, I discuss the beleaguered nature of the philosophical underpinnings of the standard practices in psychological research. Of the five assumptions he presented, the aggregate assumption results in Simpson’s paradox, a form of ecological fallacy. While the other assumptions need urgent attention, the proposals Arocha makes for addressing the aggregate assumption are still unsettled in many ways. I show that while perceptual control theory informed by the Aristotelian concept of final cause or telos allows for embracing variability as a psychological fact of human behaviour, one cannot say the same for his recommendation for the use of observation-oriented modelling (OOM) to address the aggregate assumption or to circumvent Simpson’s paradox.

Multiscale mobility explains differential associations between the gross domestic product and COVID-19 transmission in Chinese cities

Highlight In this letter, we find a Simpson’s paradox in the association between GDP and COVID-19 transmission in Chinese cities stratified by location. The differential associations in cities within and outside Hubei province can be explained by different patterns of short-range and long-range multiscale mobility from Wuhan to other cities.