scholarly journals Error curves for evaluating the quality of feature rankings

2020 ◽  
Vol 6 ◽  
pp. e310
Author(s):  
Ivica Slavkov ◽  
Matej Petković ◽  
Pierre Geurts ◽  
Dragi Kocev ◽  
Sašo Džeroski
Keyword(s):  
Predictive Models ◽  
Synthetic Data ◽  
Benchmark Problems ◽  
Feature Ranking ◽  
Ranking Algorithm ◽  
Target Variable ◽  
Ranking Algorithms ◽  
Error Measures ◽  
Nested Sets

In this article, we propose a method for evaluating feature ranking algorithms. A feature ranking algorithm estimates the importance of descriptive features when predicting the target variable, and the proposed method evaluates the correctness of these importance values by computing the error measures of two chains of predictive models. The models in the first chain are built on nested sets of top-ranked features, while the models in the other chain are built on nested sets of bottom ranked features. We investigate which predictive models are appropriate for building these chains, showing empirically that the proposed method gives meaningful results and can detect differences in feature ranking quality. This is first demonstrated on synthetic data, and then on several real-world classification benchmark problems.

RANKuser

2018 ◽  
Vol 52 (3) ◽  
pp. 329-350 ◽  
Author(s):  
Abhishek Kumar Singh ◽  
Naresh Kumar Nagwani ◽  
Sudhakar Pandey
Keyword(s):  
Question Answering ◽  
User Profile ◽  
High Volume ◽  
Ranking Algorithm ◽  
Content Type ◽  
Community Question Answering ◽  
Ranking Algorithms ◽  
Expert User ◽  
Third Stage

Purpose Recently, with a high volume of users and user’s content in Community Question Answering (CQA) sites, the quality of answers provided by users has raised a big concern. Finding the expert users can be a method to address this problem, which aims to find the suitable users (answerers) who can provide high-quality relevant answers. The purpose of this paper is to find the expert users for the newly posted questions of the CQA sites. Design/methodology/approach In this paper, a new algorithm, RANKuser, is proposed for identifying the expert users of CQA sites. The proposed RANKuser algorithm consists of three major stages. In the first stage, folksonomy relation between users, tags, and queries is established. User profile attributes, namely, reputation, tags, and badges, are also considered in folksonomy. In the second stage, expertise scores of the user are calculated based on reputation, badges, and tags. Finally, in the third stage, the expert users are identified by extracting top N users based on expertise score. Findings In this work, with the help of proposed ranking algorithm, expert users are identified for newly posted questions. In this paper, comparison of proposed user ranking algorithm (RANKuser) is also performed with other existing ranking algorithms, namely, ML-KNN, rankSVM, LDA, STM CQARank, and EV-based model using performance parameters such as hamming loss, accuracy, average precision, one error, F-measure, and normalized discounted cumulative gain. The proposed ranking method is also compared to the original ranking of CQA sites using the paired t-test. The experimental results demonstrate the effectiveness of the proposed RANKuser algorithm in comparison with the existing ranking algorithms. Originality/value This paper proposes and implements a new algorithm for expert user identification in CQA sites. By utilizing the folksonomy in CQA sites and information of user profile, this algorithm identifies the experts.

scholarly journals Search Engine-inspired Ranking Algorithm for Trading Networks

2018 ◽  
Vol 9 (3) ◽  
pp. 812
Author(s):  
Andri Mirzal
Keyword(s):  
United Nations ◽  
Web Search ◽  
Similarity Measures ◽  
Ranking Algorithm ◽  
Link Structure ◽  
Ranking Algorithms ◽  
International Trading ◽  
Trading Network ◽  
Trading Networks

<p>Ranking algorithms based on link structure of the network are well-known methods in web search engines to improve the quality of the searches. The most famous ones are PageRank and HITS. PageRank uses probability of random surfers to visit a page as the score of that page, and HITS instead of produces one score, proposes using two scores, authority and hub scores, where the authority scores describe the degree of popularity of pages and hub scores describe the quality of hyperlinks on pages. In this paper, we show the differences between WWW network and trading network, and use these differences to create a ranking algorithm for trading networks. We test our proposed method with international trading data from United Nations. The similarity measures between vectors of proposed algorithm and vector of standard measure give promising results.</p>

Simple Index to Assess the Calibration Quality of Safety Performance Functions Based on Multiple Goodness-of-Fit Metrics

2021 ◽  
pp. 036119812110088
Author(s):  
Raul E. Avelar ◽  
Karen Dixon ◽  
Boniphace Kutela ◽  
Sam Klump ◽  
Beth Wemple ◽  
...  
Keyword(s):  
Goodness Of Fit ◽  
Synthetic Data ◽  
Calibration Procedure ◽  
Safety Performance ◽  
Absolute Deviation ◽  
Data Set ◽  
Safety Database ◽  
Simple Index ◽  
Safety Performance Functions

The calibration of safety performance functions (SPFs) is a mechanism included in the Highway Safety Manual (HSM) to adjust SPFs in the HSM for use in intended jurisdictions. Critically, the quality of the calibration procedure must be assessed before using the calibrated SPFs. Multiple resources to aid practitioners in calibrating SPFs have been developed in the years following the publication of the HSM 1st edition. Similarly, the literature suggests multiple ways to assess the goodness-of-fit (GOF) of a calibrated SPF to a data set from a given jurisdiction. This paper uses the calibration results of multiple intersection SPFs to a large Mississippi safety database to examine the relations between multiple GOF metrics. The goal is to develop a sensible single index that leverages the joint information from multiple GOF metrics to assess overall quality of calibration. A factor analysis applied to the calibration results revealed three underlying factors explaining 76% of the variability in the data. From these results, the authors developed an index and performed a sensitivity analysis. The key metrics were found to be, in descending order: the deviation of the cumulative residual (CURE) plot from the 95% confidence area, the mean absolute deviation, the modified R-squared, and the value of the calibration factor. This paper also presents comparisons between the index and alternative scoring strategies, as well as an effort to verify the results using synthetic data. The developed index is recommended to comprehensively assess the quality of the calibrated intersection SPFs.

scholarly journals Benchmarking Unsupervised Outlier Detection with Realistic Synthetic Data

2021 ◽  
Vol 15 (4) ◽  
pp. 1-20
Author(s):  
Georg Steinbuss ◽  
Klemens Böhm
Keyword(s):  
Outlier Detection ◽  
Synthetic Data ◽  
Real Data ◽  
Detection Methods ◽  
Quality Of Data ◽  
Benchmark Data ◽  
Core Idea ◽  
Generic Process ◽  
Unsupervised Outlier Detection

Benchmarking unsupervised outlier detection is difficult. Outliers are rare, and existing benchmark data contains outliers with various and unknown characteristics. Fully synthetic data usually consists of outliers and regular instances with clear characteristics and thus allows for a more meaningful evaluation of detection methods in principle. Nonetheless, there have only been few attempts to include synthetic data in benchmarks for outlier detection. This might be due to the imprecise notion of outliers or to the difficulty to arrive at a good coverage of different domains with synthetic data. In this work, we propose a generic process for the generation of datasets for such benchmarking. The core idea is to reconstruct regular instances from existing real-world benchmark data while generating outliers so that they exhibit insightful characteristics. We propose and describe a generic process for the benchmarking of unsupervised outlier detection, as sketched so far. We then describe three instantiations of this generic process that generate outliers with specific characteristics, like local outliers. To validate our process, we perform a benchmark with state-of-the-art detection methods and carry out experiments to study the quality of data reconstructed in this way. Next to showcasing the workflow, this confirms the usefulness of our proposed process. In particular, our process yields regular instances close to the ones from real data. Summing up, we propose and validate a new and practical process for the benchmarking of unsupervised outlier detection.

A parameterless feature ranking algorithm based on MI

2008 ◽  
Vol 71 (7-9) ◽  
pp. 1656-1668 ◽  
Author(s):  
Jin-Jie Huang ◽  
Yun-Ze Cai ◽  
Xiao-Ming Xu
Keyword(s):  
Feature Ranking ◽  
Ranking Algorithm

scholarly journals Number of Clusters and the Quality of Hybrid Predictive Models in Analytical CRM

2014 ◽  
Vol 37 (1) ◽  
pp. 141-157 ◽  
Author(s):  
Mariusz Łapczyński ◽  
Bartłomiej Jefmański
Keyword(s):  
Predictive Models ◽  
Cluster Validity ◽  
Number Of Clusters ◽  
Model Combining ◽  
Cluster Validity Indices ◽  
Validity Indices ◽  
And Cluster Analysis ◽  
Analytical Tools ◽  
F Measure

Abstract Making more accurate marketing decisions by managers requires building effective predictive models. Typically, these models specify the probability of customer belonging to a particular category, group or segment. The analytical CRM categories refer to customers interested in starting cooperation with the company (acquisition models), customers who purchase additional products (cross- and up-sell models) or customers intending to resign from the cooperation (churn models). During building predictive models researchers use analytical tools from various disciplines with an emphasis on their best performance. This article attempts to build a hybrid predictive model combining decision trees (C&RT algorithm) and cluster analysis (k-means). During experiments five different cluster validity indices and eight datasets were used. The performance of models was evaluated by using popular measures such as: accuracy, precision, recall, G-mean, F-measure and lift in the first and in the second decile. The authors tried to find a connection between the number of clusters and models' quality.

A multi-objective evolutionary algorithm based on niche selection in solving irregular Pareto fronts

2021 ◽  
pp. 1-21
Author(s):  
Xin Li ◽  
Xiaoli Li ◽  
Kang Wang
Keyword(s):  
Evolutionary Algorithm ◽  
Optimization Problems ◽  
Optimal Solution ◽  
Benchmark Problems ◽  
Multi Objective Optimization ◽  
Multi Objective ◽  
Environmental Selection ◽  
Optimal Solution Set ◽  
Pareto Fronts

The key characteristic of multi-objective evolutionary algorithm is that it can find a good approximate multi-objective optimal solution set when solving multi-objective optimization problems(MOPs). However, most multi-objective evolutionary algorithms perform well on regular multi-objective optimization problems, but their performance on irregular fronts deteriorates. In order to remedy this issue, this paper studies the existing algorithms and proposes a multi-objective evolutionary based on niche selection to deal with irregular Pareto fronts. In this paper, the crowding degree is calculated by the niche method in the process of selecting parents when the non-dominated solutions converge to the first front, which improves the the quality of offspring solutions and which is beneficial to local search. In addition, niche selection is adopted into the process of environmental selection through considering the number and the location of the individuals in its niche radius, which improve the diversity of population. Finally, experimental results on 23 benchmark problems including MaF and IMOP show that the proposed algorithm exhibits better performance than the compared MOEAs.

scholarly journals Prognostic Factors for Development of Subsequent Metastases in Localized Osteosarcoma: A Systematic Review and Identification of Literature Gaps

Sarcoma ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-7 ◽  
Author(s):  
Patrick Basile ◽  
Emily Greengard ◽  
Brenda Weigel ◽  
Logan Spector
Keyword(s):  
Systematic Review ◽  
Prognostic Factors ◽  
Predictive Models ◽  
Pulmonary Metastases ◽  
Tumor Location ◽  
Evidence Based ◽  
Review Of The Literature ◽  
Histologic Response ◽  
Exclusion Criteria

Aim. To investigate prognostic factors in pediatric and young adult patients with localized osteosarcoma that could predict the development of subsequent pulmonary metastases and lead to an ability to risk-stratify therapy. We performed a systematic review of the literature published since January 1990 to establish common evidence-based prognostic factors. Methods. PubMed and Embase searches (Jan 1990–Aug 2018) were performed. Two reviewers independently selected papers for patients with localized osteosarcoma with subsequent metastatic development and then reviewed for quality of methods and prognostic factors. Results. Database searches yielded 216 unique results. After screening, 27 full-text articles were studied in depth, with 9 items fulfilling predetermined inclusion and exclusion criteria. Age, tumor location, tumor size/volume, and histologic response carried independent prognostic value in the majority of the studies. Conclusions. Several prognostic factors seemed to be consistent amongst the studies, but the heterogeneity and smaller sizes of the study populations made pooling of results difficult. Standardization of larger patient populations and consistent definitions/cutoffs for prognostic factors are needed to further assess for consistent prognostic factors and potential predictive models to be developed.

scholarly journals SignRank: A Novel Random Walking Based Ranking Algorithm in Signed Networks

2019 ◽  
Vol 2019 ◽  
pp. 1-8 ◽  
Author(s):  
Cong Wan ◽  
Yanhui Fang ◽  
Cong Wang ◽  
Yanxia Lv ◽  
Zejie Tian ◽  
...  
Keyword(s):  
Social Networks ◽  
Social Network ◽  
Positive Emotion ◽  
Ranking Algorithm ◽  
Random Walking ◽  
Signed Networks ◽  
Ranking Algorithms ◽  
Negative Score ◽  
Positive Score ◽  
Signed Network

Social networks have become an indispensable part of modern life. Signed networks, a class of social network with positive and negative edges, are becoming increasingly important. Many social networks have adopted the use of signed networks to model like (trust) or dislike (distrust) relationships. Consequently, how to rank nodes from positive and negative views has become an open issue of social network data mining. Traditional ranking algorithms usually separate the signed network into positive and negative graphs so as to rank positive and negative scores separately. However, much global information of signed network gets lost during the use of such methods, e.g., the influence of a friend’s enemy. In this paper, we propose a novel ranking algorithm that computes a positive score and a negative score for each node in a signed network. We introduce a random walking model for signed network which considers the walker has a negative or positive emotion. The steady state probability of the walker visiting a node with negative or positive emotion represents the positive score or negative score. In order to evaluate our algorithm, we use it to solve sign prediction problem, and the result shows that our algorithm has a higher prediction accuracy compared with some well-known ranking algorithms.

Sign in / Sign up

Export Citation Format

Share Document