A model-based approach for text clustering with outlier detection

2016 ◽  
Author(s):  
Jianhua Yin ◽  
Jianyong Wang
Keyword(s):  
Outlier Detection ◽  
Text Clustering ◽  
Model Based

Failure analysis for model-based organ segmentation using outlier detection

2014 ◽  
Author(s):  
Axel Saalbach ◽  
Irina Wächter Stehle ◽  
Cristian Lorenz ◽  
Jürgen Weese
Keyword(s):  
Failure Analysis ◽  
Outlier Detection ◽  
Model Based

scholarly journals X-DMM: Fast and Scalable Model Based Text Clustering

2019 ◽  
Vol 33 ◽  
pp. 4197-4204 ◽  
Author(s):  
Linwei Li ◽  
Liangchen Guo ◽  
Zhenying He ◽  
Yinan Jing ◽  
X. Sean Wang
Keyword(s):  
Time Complexity ◽  
Large Scale ◽  
Dirichlet Distribution ◽  
Average Length ◽  
Scale Up ◽  
Text Clustering ◽  
Number Of Clusters ◽  
Model Based ◽  
Model Training ◽  
Real World Datasets

Text clustering is a widely studied problem in the text mining domain. The Dirichlet Multinomial Mixture (DMM) model based clustering algorithms have shown good performance to cope with high dimensional sparse text data, obtaining reasonable results in both clustering accuracy and computational efficiency. However, the time complexity of DMM model training is proportional to the average document length and the number of clusters, making it inefficient for scaling up to long text and large corpora, which is common in realworld applications such as documents organization, retrieval and recommendation. In this paper, we leverage a symmetric prior setting for Dirichlet distribution, and build indices to decrease the time complexity of the sampling-based training for DMM from O(K∗L) to O(K∗U), where K is the number of clusters, L the average length of document, and U the average number of unique words in each document. We introduce a Metropolis-Hastings sampling algorithm, which further reduces the sampling time complexity from O(K∗U) to O(U) in the nearly-to-convergence training stages. Moreover, we also parallelize the DMM model training to obtain a further acceleration by using an uncollapsed Gibbs sampler. We combine all these optimizations into a highly efficient implementation, called X-DMM, which enables the DMM model to scale up for long and large-scale text clustering. We evaluate the performance of X-DMM on several real world datasets, and the experimental results show that XDMM achieves substantial speed up compared with existing state-of-the-art algorithms without clustering accuracy degradation.

Sign in / Sign up

Export Citation Format

Share Document