scholarly journals A Topical Word Embeddings for Text Classification

2018 ◽  
Author(s):  
João Marcos Carvalho Lima ◽  
José Everardo Bessa Maia
Keyword(s):  
Text Classification ◽  
Text Categorization ◽  
Topic Model ◽  
State Of The Art ◽  
Bag Of Words ◽  
Word Embeddings ◽  
Document Representation ◽  
Linear Classifier ◽  
Low Dimensionality ◽  
Nonlinear Classifier

This paper presents an approach that uses topic models based on LDA to represent documents in text categorization problems. The document representation is achieved through the cosine similarity between document embeddings and embeddings of topic words, creating a Bag-of-Topics (BoT) variant. The performance of this approach is compared against those of two other representations: BoW (Bag-of-Words) and Topic Model, both based on standard tf-idf. Also, to reveal the effect of the classifier, we compared the performance of the nonlinear classifier SVM against that of the linear classifier Naive Bayes, taken as baseline. To evaluate the approach we use two bases, one multi-label (RCV-1) and another single-label (20 Newsgroup). The model presents significant results with low dimensionality when compared to the state of the art.

scholarly journals Dataless Short Text Classification Based on Biterm Topic Model and Word Embeddings

2020 ◽  
Author(s):  
Yi Yang ◽  
Hongan Wang ◽  
Jiaqi Zhu ◽  
Yunkun Wu ◽  
Kailong Jiang ◽  
...  
Keyword(s):  
Text Classification ◽  
Classification Accuracy ◽  
Topic Model ◽  
State Of The Art ◽  
The Internet ◽  
Word Embeddings ◽  
Word Similarity ◽  
Short Text ◽  
Category Word ◽  
Novel Model

Dataless text classification has attracted increasing attentions recently. It only needs very few seed words of each category to classify documents, which is much cheaper than supervised text classification that requires massive labeling efforts. However, most of existing models pay attention to long texts, but get unsatisfactory performance on short texts, which have become increasingly popular on the Internet. In this paper, we at first propose a novel model named Seeded Biterm Topic Model (SeedBTM) extending BTM to solve the problem of dataless short text classification with seed words. It takes advantage of both word co-occurrence information in the topic model and category-word similarity from widely used word embeddings as the prior topic-in-set knowledge. Moreover, with the same approach, we also propose Seeded Twitter Biterm Topic Model (SeedTBTM), which extends Twitter-BTM and utilizes additional user information to achieve higher classification accuracy. Experimental results on five real short-text datasets show that our models outperform the state-of-the-art methods, and especially perform well when the categories are overlapping and interrelated.

On Document Representation and Term Weights in Text Classification

2010 ◽  
pp. 1-22 ◽  
Author(s):  
Ying Liu
Keyword(s):  
Text Classification ◽  
Semantic Information ◽  
Weighting Scheme ◽  
Bag Of Words ◽  
Document Representation ◽  
Term Weighting ◽  
Word Sequence ◽  
Sentence Level ◽  
Sequence Method ◽  
Classic Approach

In the automated text classification, a bag-of-words representation followed by the tfidf weighting is the most popular approach to convert the textual documents into various numeric vectors for the induction of classifiers. In this chapter, we explore the potential of enriching the document representation with the semantic information systematically discovered at the document sentence level. The salient semantic information is searched using a frequent word sequence method. Different from the classic tfidf weighting scheme, a probability based term weighting scheme which directly reflect the term’s strength in representing a specific category has been proposed. The experimental study based on the semantic enriched document representation and the newly proposed probability based term weighting scheme has shown a significant improvement over the classic approach, i.e., bag-of-words plus tfidf, in terms of Fscore. This study encourages us to further investigate the possibility of applying the semantic enriched document representation over a wide range of text based mining tasks.

scholarly journals A Novel Neural Network-Based Method for Medical Text Classification

2019 ◽  
Vol 11 (12) ◽  
pp. 255 ◽  
Author(s):  
Li Qing ◽  
Weng Linhong ◽  
Ding Xuehai
Keyword(s):  
Neural Network ◽  
Text Classification ◽  
Text Categorization ◽  
Medical Literature ◽  
Clinical Information ◽  
Specific Area ◽  
Attention Mechanism ◽  
Document Representation ◽  
Medical Texts ◽  
Medical Text

Medical text categorization is a specific area of text categorization. Classification for medical texts is considered a special case of text classification. Medical text includes medical records and medical literature, both of which are important clinical information resources. However, medical text contains complex medical vocabularies, medical measures, which has problems with high-dimensionality and data sparsity, so text classification in the medical domain is more challenging than those in other general domains. In order to solve these problems, this paper proposes a unified neural network method. In the sentence representation, the convolutional layer extracts features from the sentence and a bidirectional gated recurrent unit (BIGRU) is used to access both the preceding and succeeding sentence features. An attention mechanism is employed to obtain the sentence representation with the important word weights. In the document representation, the method uses the BIGRU to encode the sentences, which is obtained in sentence representation and then decode it through the attention mechanism to get the document representation with important sentence weights. Finally, a category of medical text is obtained through a classifier. Experimental verifications are conducted on four medical text datasets, including two medical record datasets and two medical literature datasets. The results clearly show that our method is effective.

scholarly journals Large scale hierarchical text classification

2015 ◽  
Author(s):  
Άρης Κοσμόπουλος
Keyword(s):  
Principal Component Analysis ◽  
Text Classification ◽  
Large Scale ◽  
Principal Component ◽  
Component Analysis ◽  
Bag Of Words ◽  
Word Embeddings ◽  
Medical Text ◽  
Hierarchical Text Classification

Οι ιεραρχίες χρησιμοποιούνται όλο και πιο συχνά στην την οργάνωση κειμένων και η χρήση αυτή είναι ακόμη πιο συχνή στο διαδίκτυο. Οι κατάλογοι ιστοσελίδων, όπως το Yahoo Directory και το Dmoz Directory, είναι τέτοια τυπικά παραδείγματα. Μαζί με την συχνή χρήση τους όμως προκύπτει και η ανάγκη για αυτοματοποιημένους τρόπους ταξινόμησης των νέων κειμένων στις κατηγορίες των ιεραρχιών αυτών. Σε αυτή τη διατριβή, ονομάζουμε το πρόβλημα αυτό "μεγάλης κλίμακας Ιεραρχική κατηγοριοποίηση κειμένων". Είναι μεγάλης κλίμακας, γιατί οι κατηγορίες είναι χιλιάδες και τα κείμενα μπορεί να είναι από εκατοντάδες χιλιάδες μέχρι και εκατομμύρια. Είναι επίσης ιεραρχικό επειδή οι κατηγορίες συν΄δεονται μεταξύ τους με σχέσεις γονέα-πατέρα. Ένα σημαντικό θέμα στην ιεραρχική κατηγοριοποίηση είναι η αξιολόγηση διαφορετικών αλγορίθμων κατηγοριοποίησης, που είναι ακόμη πιο έντονο λόγο της ύπαρξης της ιεραρχίας. Διάφορα ιεραρχικά μέτρα έχουν προταθεί στο παρελθόν, αλλά χωρίς να προσφέρουν ένα ενοποιημένο τρόπο εποπτείας του προβλήματος. Σε αυτή τη διατριβή, μελετούμε το πρόβλημα της αξιολόγησης στην ιεραρχική κατηγοριοποίηση, αναλύοντας τα βασικά στοιχεία των υπαρχόντων ιεραρχικών μέτρων. Επίσης διαχωρίζουμε τα υπάρχοντα ιεραρχικά μέτρα σε δυο εναλλακτικά γενικά μοντέλα και προτείνουμε δυο καινοτόμα μέτρα για κάθε μοντέλο. Τα υπάρχοντα και τα προτεινόμενα μέτρα δοκιμάζονται σε τρία μεγάλα σύνολα δεδομένων κατηγοριοποίησης κειμένων. Τα αποτελέσματα των πειραμάτων δείχνουν τους περιορισμούς των υπαρχόντων μέτρων και το πως τα νέα προτεινόμενα μέτρα ξεπερνούν αυτούς τους περιορισμούς. Στη συνέχεια επικεντρωνόμαστε στην απλούστερη μορφή ιεραρχικής κατηγοριοποίησης όπου κάθε κείμενο ανήκει σε μόνο μία κατηγορία και η ιεραρχία έχει μορφή δένδρου. Η πιο συνηθισμένη μορφή ιεραρχικής κατηγοριοποίησης είναι αυτή του Cascade, στην οποία διατρέχεται η ιεραρχία από τη ρίζα του δένδρου ως το προτεινόμενο φύλλο. Για να πραγματοποιηθεί αυτή η διαδικασία, πρέπει να εκπαιδευτεί ένας ταξινομητής σε κάθε κόμβο του δένδρου, αλλά στα πιο ψηλά επίπεδα ο αριθμός των χαρακτηριστικών μπορεί να γίνει απαγορευτικά υψηλός. Για αυτό και είναι επιθυμητή η μείωση της διαστασιμότητας του χώρου των χαρακτηριστικών σε αυτά τα επίπεδα. Δεδομένου ότι η πιο ευρέος διαδεδομένη μέθοδος μείωσης χαρακτηριστικών είναι το Principal Component Analysis (PCA), εξετάζουμε τη χρήση του στο Cascade μελετώντας την επίδραση του στο υπολογιστικό κόστος αλλά και την ακρίβεια των ταξινομικών. Επίσης προτείνουμε έναν εναλλακτικό τρόπο πιθανοτικού Cascade ο οποίος κάνοντας καλύτερη χρήση των πιθανοτήτων των ταξινομητών επιτυγχάνει καλύτερα αποτελέσματα σε σχέση με το παραδοσιακό Cascade. Τέλος, εξετάζουμε ένα πιο πολύπλοκο πρόβλημα, γνωστό ως βιοϊατρική σημασιολογική ταξινόμηση όπου βιοϊατρικά κείμενα πρέπει να ταξινομηθούν σε κατηγορίες που ανήκουν σε μια μεγάλη βιοϊατρική ιεραρχία. Το πρόβλημα αυτό είναι πιο πολύπλοκο διότι η ιεραρχία είναι κατευθυνόμενος γράφος και όχι απλά δένδρο, ενώ κάθε κείμενο μπορεί να ανήκει σε πολλές κατηγορίες η οποίες μάλιστα μπορεί να μην είναι απαραίτητα φύλλα του γράφου. Σε αυτό το πρόβλημα, εξετάζουμε της χρήση πυκνών διανυσμάτων λέξεων (word embeddings) ως ένα τρόπο για μείωση της διαστασημότητας των χαρακτηριστικών. Εξετάζουμε διάφορες προσεγγίσεις για να περάσουμε από τα διανύσματα λέξεων σε διανύσματα κειμένων και προτείνουμε μια απλή διαδικασία με χρήση κεντροειδούς η οποία είναι κατάλληλη για το πρόβλημα. Επίσης δείχνουμε πως η υιοθέτηση αυτής της προσέγγισης κάνει το πρόβλημα της μεγάλης κλίμακας ιεραρχικής κατηγοριοποίησης πολύ πιο κλιμακώσιμο, χωρίς να υστερεί σε ακρίβεια σε σχέση με τη συνηθισμένη προσέγγιση bag-of-words. Στα πειράματά μας εξετάζουμε τη χρήση ιεραρχικών και μη ιεραρχικών ταξινομητών κ-κοντινότερων-γειτόνων και μελετάμε την επίδραση των διαφόρων παραμέτρων τους. Επίσης παρουσιάζουμε ένα υψηλής ακρίβειας σύστημα που συνδυάζεται με το ευρέος χρησιμοποιημένο Medical Text Indexer (MTI) σύστημα της Εθνικής Βιβλιοθήκης της Ιατρικής με στόχο τη βελτίωση των προβλέψεών του.

scholarly journals Multivariate Gaussian Document Representation from Word Embeddings for Text Categorization

2017 ◽  
Author(s):  
Giannis Nikolentzos ◽  
Polykarpos Meladianos ◽  
Francois Rousseau ◽  
Yannis Stavrakas ◽  
Michalis Vazirgiannis
Keyword(s):  
Text Categorization ◽  
Word Embeddings ◽  
Document Representation ◽  
Multivariate Gaussian

scholarly journals Emotionally charged text classification with deep learning and sentiment semantic

2021 ◽  
Author(s):  
Jeow Li Huan ◽  
Arif Ahmed Sekh ◽  
Chai Quek ◽  
Dilip K. Prasad
Keyword(s):  
Language Processing ◽  
Text Classification ◽  
Classification Accuracy ◽  
State Of The Art ◽  
Document Representation ◽  
Classical Technique ◽  
Text Classifiers ◽  
Vector Sequences ◽  
Fully Connected ◽  
Better Than

AbstractText classification is one of the widely used phenomena in different natural language processing tasks. State-of-the-art text classifiers use the vector space model for extracting features. Recent progress in deep models, recurrent neural networks those preserve the positional relationship among words achieve a higher accuracy. To push text classification accuracy even higher, multi-dimensional document representation, such as vector sequences or matrices combined with document sentiment, should be explored. In this paper, we show that documents can be represented as a sequence of vectors carrying semantic meaning and classified using a recurrent neural network that recognizes long-range relationships. We show that in this representation, additional sentiment vectors can be easily attached as a fully connected layer to the word vectors to further improve classification accuracy. On the UCI sentiment labelled dataset, using the sequence of vectors alone achieved an accuracy of 85.6%, which is better than 80.7% from ridge regression classifier—the best among the classical technique we tested. Additional sentiment information further increases accuracy to 86.3%. On our suicide notes dataset, the best classical technique—the Naíve Bayes Bernoulli classifier, achieves accuracy of 71.3%, while our classifier, incorporating semantic and sentiment information, exceeds that at 75% accuracy.

scholarly journals Application of the Polyhedral Conic Functions Method in the Text Classification and Comparative Analysis

2018 ◽  
Vol 2018 ◽  
pp. 1-11
Author(s):  
Nur Uylaş Satı ◽  
Burak Ordin
Keyword(s):  
Text Classification ◽  
Numerical Experiments ◽  
Text Categorization ◽  
Cross Validation ◽  
State Of The Art ◽  
Inequality Constraints ◽  
Online Information ◽  
Direct Proportion ◽  
Art Methods ◽  
Tenfold Cross Validation

In direct proportion to the heavy increase of online information data, the attention to text categorization (classification) has also increased. In text categorization problem, namely, text classification, the goal is to classify the documents into predefined classes (categories or labels). Recently various methods in data mining have been experienced for text classification in literature except polyhedral conic function (PCF) methods. In this paper, PCFs are used to classify the documents. The separation algorithms via PCFs which include linear programming subproblems with inequality constraints are presented. Numerical experiments are done on real-world text datasets. Comparisons are made between state-of-the-art methods by presenting obtained tenfold cross-validation results, accuracy values, and running times in tables. The results verify that in text classification PCF methods are as effective in terms of accuracy values as state-of-the-art methods.

scholarly journals Near-Lossless Binarization of Word Embeddings

2019 ◽  
Vol 33 ◽  
pp. 7104-7111 ◽  
Author(s):  
Julien Tissier ◽  
Christophe Gravier ◽  
Amaury Habrard
Keyword(s):  
Sentiment Analysis ◽  
Semantic Similarity ◽  
Text Classification ◽  
Semantic Information ◽  
State Of The Art ◽  
Floating Point ◽  
Word Embeddings ◽  
Binary Vectors ◽  
Starting Point ◽  
Memory Footprint

Word embeddings are commonly used as a starting point in many NLP models to achieve state-of-the-art performances. However, with a large vocabulary and many dimensions, these floating-point representations are expensive both in terms of memory and calculations which makes them unsuitable for use on low-resource devices. The method proposed in this paper transforms real-valued embeddings into binary embeddings while preserving semantic information, requiring only 128 or 256 bits for each vector. This leads to a small memory footprint and fast vector operations. The model is based on an autoencoder architecture, which also allows to reconstruct original vectors from the binary ones. Experimental results on semantic similarity, text classification and sentiment analysis tasks show that the binarization of word embeddings only leads to a loss of ∼2% in accuracy while vector size is reduced by 97%. Furthermore, a top-k benchmark demonstrates that using these binary vectors is 30 times faster than using real-valued vectors.

scholarly journals Improving Entity Linking by Modeling Latent Entity Type Information

2020 ◽  
Vol 34 (05) ◽  
pp. 7529-7537 ◽  
Author(s):  
Shuang Chen ◽  
Jinpeng Wang ◽  
Feng Jiang ◽  
Chin-Yew Lin
Keyword(s):  
State Of The Art ◽  
Similarity Score ◽  
The State ◽  
Local Context ◽  
Bag Of Words ◽  
Word Embeddings ◽  
Entity Linking ◽  
Context Model ◽  
Type Information ◽  
Experiment Analysis

Existing state of the art neural entity linking models employ attention-based bag-of-words context model and pre-trained entity embeddings bootstrapped from word embeddings to assess topic level context compatibility. However, the latent entity type information in the immediate context of the mention is neglected, which causes the models often link mentions to incorrect entities with incorrect type. To tackle this problem, we propose to inject latent entity type information into the entity embeddings based on pre-trained BERT. In addition, we integrate a BERT-based entity similarity score into the local context model of a state-of-the-art model to better capture latent entity type information. Our model significantly outperforms the state-of-the-art entity linking models on standard benchmark (AIDA-CoNLL). Detailed experiment analysis demonstrates that our model corrects most of the type errors produced by the direct baseline.

Deep Learning for text in limted data settings

2020 ◽  
Author(s):  
Pathikkumar Patel ◽  
Bhargav Lad ◽  
Jinan Fiaidhi
Keyword(s):  
Machine Learning ◽  
Time Series ◽  
Deep Learning ◽  
Sentiment Analysis ◽  
Transfer Learning ◽  
Text Classification ◽  
State Of The Art ◽  
Time Series Forecasting ◽  
Text Data ◽  
Performance Levels

During the last few years, RNN models have been extensively used and they have proven to be better for sequence and text data. RNNs have achieved state-of-the-art performance levels in several applications such as text classification, sequence to sequence modelling and time series forecasting. In this article we will review different Machine Learning and Deep Learning based approaches for text data and look at the results obtained from these methods. This work also explores the use of transfer learning in NLP and how it affects the performance of models on a specific application of sentiment analysis.

Sign in / Sign up

Export Citation Format

Share Document