scholarly journals A user-centric evaluation framework for recommender systems

2011 ◽  
Author(s):  
Pearl Pu ◽  
Li Chen ◽  
Rong Hu
Keyword(s):  
Recommender Systems ◽  
Evaluation Framework ◽  
User Centric

A Characterisation and Framework for User-Centric Factors in Evaluation Methods for Recommender Systems

2017 ◽  
Vol 6 (1) ◽  
pp. 1-16 ◽  
Author(s):  
Tatenda D. Kavu ◽  
Kuda Dube ◽  
Peter G. Raeth ◽  
Gilford T. Hapanyengwi
Keyword(s):  
Conceptual Framework ◽  
Recommender Systems ◽  
User Experience ◽  
User Satisfaction ◽  
Optimal Solution ◽  
User Interaction ◽  
Evaluation Methods ◽  
User Preferences ◽  
Evaluation Framework ◽  
User Centric

Researchers have worked on-finding e-commerce recommender systems evaluation methods that contribute to an optimal solution. However, existing evaluations methods lack the assessment of user-centric factors such as buying decisions, user experience and user interactions resulting in less than optimum recommender systems. This paper investigates the problem of adequacy of recommender systems evaluation methods in relation to user-centric factors. Published work has revealed limitations of existing evaluation methods in terms of evaluating user satisfaction. This paper characterizes user-centric evaluation factors and then propose a user-centric evaluation conceptual framework to identify and expose a gap within literature. The researchers used an integrative review approach to formulate both the characterization and the conceptual framework for investigation. The results reveal a need to come up with a holistic evaluation framework that combines system-centric and user-centric evaluation methods as well as formulating computational user-centric evaluation methods. The conclusion reached is that, evaluation methods for e-commerce recommender systems lack full assessment of vital factors such as: user interaction, user experience and purchase decisions. A full consideration of these factors during evaluation will give birth to new types of recommender systems that predict user preferences using user decision-making process profiles, and that will enhance user experience and increase revenue in the long run.

Clustering Approaches for Top-k Recommender Systems

2019 ◽  
Vol 28 (05) ◽  
pp. 1950019 ◽  
Author(s):  
Nicolás Torres ◽  
Marcelo Mendoza
Keyword(s):  
Recommender Systems ◽  
Recommender System ◽  
Large Scale ◽  
Empirical Studies ◽  
Cluster Structure ◽  
Parallel Architecture ◽  
Evaluation Framework ◽  
Training Data ◽  
Cluster Solution ◽  
Novel Approach

Clustering-based recommender systems bound the seek of similar users within small user clusters providing fast recommendations in large-scale datasets. Then groups can naturally be distributed into different data partitions scaling up in the number of users the recommender system can handle. Unfortunately, while the number of users and items included in a cluster solution increases, the performance in terms of precision of a clustering-based recommender system decreases. We present a novel approach that introduces a cluster-based distance function used for neighborhood computation. In our approach, clusters generated from the training data provide the basis for neighborhood selection. Then, to expand the search of relevant users, we use a novel measure that can exploit the global cluster structure to infer cluster-outside user’s distances. Empirical studies on five widely known benchmark datasets show that our proposal is very competitive in terms of precision, recall, and NDCG. However, the strongest point of our method relies on scalability, reaching speedups of 20× in a sequential computing evaluation framework and up to 100× in a parallel architecture. These results show that an efficient implementation of our cluster-based CF method can handle very large datasets providing also good results in terms of precision, avoiding the high computational costs involved in the application of more sophisticated techniques.

scholarly journals Symmetric Metric Learning with Adaptive Margin for Recommendation

2020 ◽  
Vol 34 (04) ◽  
pp. 4634-4641
Author(s):  
Mingming Li ◽  
Shuai Zhang ◽  
Fuqing Zhu ◽  
Wanhui Qian ◽  
Liangjun Zang ◽  
...  
Keyword(s):  
Recommender Systems ◽  
Metric Space ◽  
State Of The Art ◽  
Metric Learning ◽  
Current User ◽  
Experimental Results ◽  
Negative Item ◽  
Positive Item ◽  
User Centric ◽  
The Impact

Metric learning based methods have attracted extensive interests in recommender systems. Current methods take the user-centric way in metric space to ensure the distance between user and negative item to be larger than that between the current user and positive item by a fixed margin. While they ignore the relations among positive item and negative item. As a result, these two items might be positioned closely, leading to incorrect results. Meanwhile, different users usually have different preferences, the fixed margin used in those methods can not be adaptive to various user biases, and thus decreases the performance as well. To address these two problems, a novel Symmetic Metric Learning with adaptive margin (SML) is proposed. In addition to the current user-centric metric, it symmetically introduces a positive item-centric metric which maintains closer distance from positive items to user, and push the negative items away from the positive items at the same time. Moreover, the dynamically adaptive margins are well trained to mitigate the impact of bias. Experimental results on three public recommendation datasets demonstrate that SML produces a competitive performance compared with several state-of-the-art methods.

scholarly journals A Guideline-Based Approach for Assisting with the Reproducibility of Experiments in Recommender Systems Evaluation

2019 ◽  
Vol 28 (08) ◽  
pp. 1960011
Author(s):  
Nikolaos Polatidis ◽  
Elias Pimenidis ◽  
Andrew Fish ◽  
Stelios Kapetanakis
Keyword(s):  
Information Retrieval ◽  
Recommender Systems ◽  
Predictive Accuracy ◽  
Evaluation Framework ◽  
Real Dataset ◽  
Design And Implementation ◽  
Comparison Of Results ◽  
Recommendation Algorithms ◽  
New Algorithms ◽  
Optimal Settings

Recommender systems’ evaluation is usually based on predictive accuracy and information retrieval metrics, with better scores meaning recommendations are of higher quality. However, new algorithms are constantly developed and the comparison of results of algorithms within an evaluation framework is difficult since different settings are used in the design and implementation of experiments. In this paper, we propose a guidelines-based approach that can be followed to reproduce experiments and results within an evaluation framework. We have evaluated our approach using a real dataset, and well-known recommendation algorithms and metrics; to show that it can be difficult to reproduce results if certain settings are missing, thus resulting in more evaluation cycles required to identify the optimal settings.

scholarly journals Sequeval: An Offline Evaluation Framework for Sequence-Based Recommender Systems

Information ◽  
2019 ◽  
Vol 10 (5) ◽  
pp. 174 ◽  
Author(s):  
Diego Monti ◽  
Enrico Palumbo ◽  
Giuseppe Rizzo ◽  
Maurizio Morisio
Keyword(s):  
Recommender Systems ◽  
Random Fields ◽  
Conditional Random Fields ◽  
State Of The Art ◽  
Focal Point ◽  
Objective Evaluation ◽  
Lessons Learned ◽  
Evaluation Framework ◽  
Definition Of ◽  
The Right

Recommender systems have gained a lot of popularity due to their large adoption in various industries such as entertainment and tourism. Numerous research efforts have focused on formulating and advancing state-of-the-art of systems that recommend the right set of items to the right person. However, these recommender systems are hard to compare since the published evaluation results are computed on diverse datasets and obtained using different methodologies. In this paper, we researched and prototyped an offline evaluation framework called Sequeval that is designed to evaluate recommender systems capable of suggesting sequences of items. We provide a mathematical definition of such sequence-based recommenders, a methodology for performing their evaluation, and the implementation details of eight metrics. We report the lessons learned using this framework for assessing the performance of four baselines and two recommender systems based on Conditional Random Fields (CRF) and Recurrent Neural Networks (RNN), considering two different datasets. Sequeval is publicly available and it aims to become a focal point for researchers and practitioners when experimenting with sequence-based recommender systems, providing comparable and objective evaluation results.

Item Familiarity as a Possible Confounding Factor in User-Centric Recommender Systems Evaluation

i-com ◽  
2015 ◽  
Vol 14 (1) ◽  
pp. 29-39 ◽  
Author(s):  
Dietmar Jannach ◽  
Lukas Lerche ◽  
Michael Jugovac
Keyword(s):  
Recommender Systems ◽  
User Satisfaction ◽  
User Study ◽  
Predictive Accuracy ◽  
Confounding Factor ◽  
Academic Research ◽  
Strongly Correlated ◽  
Quality Factors ◽  
User Centric ◽  
Assess Quality

AbstractUser studies play an important role in academic research in the field of recommender systems as they allow us to assess quality factors other than the predictive accuracy of the underlying algorithms. User satisfaction is one such factor that is often evaluated in laboratory settings and in many experimental designs one task of the participants is to assess the suitability of the system-generated recommendations. The effort required by the user to make such an assessment can, however, depend on the user’s familiarity with the presented items and directly impact on the reported user satisfaction. In this paper, we report the results of a preliminary recommender systems user study using Mechanical Turk, which indicates that item familiarity is strongly correlated with overall satisfaction.

Sign in / Sign up

Export Citation Format

Share Document