scholarly journals Towards Robust Text Classification with Semantics-Aware Recurrent Neural Architecture

2019 ◽  
Vol 1 (2) ◽  
pp. 575-589 ◽  
Author(s):  
Blaž Škrlj ◽  
Jan Kralj ◽  
Nada Lavrač ◽  
Senja Pollak
Keyword(s):  
Text Mining ◽  
Language Processing ◽  
Text Classification ◽  
Deep Neural Networks ◽  
Semantic Knowledge ◽  
Text Documents ◽  
Neural Architecture ◽  
Classification Tasks ◽  
And Gender ◽  
Semantic Resources

Deep neural networks are becoming ubiquitous in text mining and natural language processing, but semantic resources, such as taxonomies and ontologies, are yet to be fully exploited in a deep learning setting. This paper presents an efficient semantic text mining approach, which converts semantic information related to a given set of documents into a set of novel features that are used for learning. The proposed Semantics-aware Recurrent deep Neural Architecture (SRNA) enables the system to learn simultaneously from the semantic vectors and from the raw text documents. We test the effectiveness of the approach on three text classification tasks: news topic categorization, sentiment analysis and gender profiling. The experiments show that the proposed approach outperforms the approach without semantic knowledge, with highest accuracy gain (up to 10%) achieved on short document fragments.

scholarly journals Explicit Interaction Model towards Text Classification

2019 ◽  
Vol 33 ◽  
pp. 6359-6366 ◽  
Author(s):  
Cunxiao Du ◽  
Zhaozheng Chen ◽  
Fuli Feng ◽  
Lei Zhu ◽  
Tian Gan ◽  
...  
Keyword(s):  
Language Processing ◽  
Text Classification ◽  
Deep Neural Networks ◽  
Interaction Mechanism ◽  
Interaction Model ◽  
Classification Task ◽  
Fine Grained ◽  
Word Level ◽  
Benchmark Datasets ◽  
Classification Tasks

Text classification is one of the fundamental tasks in natural language processing. Recently, deep neural networks have achieved promising performance in the text classification task compared to shallow models. Despite of the significance of deep models, they ignore the fine-grained (matching signals between words and classes) classification clues since their classifications mainly rely on the text-level representations. To address this problem, we introduce the interaction mechanism to incorporate word-level matching signals into the text classification task. In particular, we design a novel framework, EXplicit interAction Model (dubbed as EXAM), equipped with the interaction mechanism. We justified the proposed approach on several benchmark datasets including both multilabel and multi-class text classification tasks. Extensive experimental results demonstrate the superiority of the proposed method. As a byproduct, we have released the codes and parameter settings to facilitate other researches.

scholarly journals Universal Adversarial Attack via Conditional Sampling for Text Classification

2021 ◽  
Vol 11 (20) ◽  
pp. 9539
Author(s):  
Yu Zhang ◽  
Kun Shao ◽  
Junan Yang ◽  
Hui Liu
Keyword(s):  
Language Processing ◽  
Text Classification ◽  
Deep Neural Networks ◽  
High Quality ◽  
The Face ◽  
Adversarial Examples ◽  
Specific Prediction ◽  
Novel Method ◽  
Adversarial Attack ◽  
Classification Tasks

Despite deep neural networks (DNNs) having achieved impressive performance in various domains, it has been revealed that DNNs are vulnerable in the face of adversarial examples, which are maliciously crafted by adding human-imperceptible perturbations to an original sample to cause the wrong output by the DNNs. Encouraged by numerous researches on adversarial examples for computer vision, there has been growing interest in designing adversarial attacks for Natural Language Processing (NLP) tasks. However, the adversarial attacking for NLP is challenging because text is discrete data and a small perturbation can bring a notable shift to the original input. In this paper, we propose a novel method, based on conditional BERT sampling with multiple standards, for generating universal adversarial perturbations: input-agnostic of words that can be concatenated to any input in order to produce a specific prediction. Our universal adversarial attack can create an appearance closer to natural phrases and yet fool sentiment classifiers when added to benign inputs. Based on automatic detection metrics and human evaluations, the adversarial attack we developed dramatically reduces the accuracy of the model on classification tasks, and the trigger is less easily distinguished from natural text. Experimental results demonstrate that our method crafts more high-quality adversarial examples as compared to baseline methods. Further experiments show that our method has high transferability. Our goal is to prove that adversarial attacks are more difficult to detect than previously thought and enable appropriate defenses.

scholarly journals Language Semantics Interpretation with an Interaction-Based Recurrent Neural Network

2021 ◽  
Vol 3 (4) ◽  
pp. 922-945
Author(s):  
Shaw-Hwa Lo ◽  
Yiqiao Yin
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Language Processing ◽  
Text Classification ◽  
Search Algorithm ◽  
Greedy Search ◽  
Text Documents ◽  
Engineering Technique ◽  
Language Semantics ◽  
Sequential Models

Text classification is a fundamental language task in Natural Language Processing. A variety of sequential models are capable of making good predictions, yet there is a lack of connection between language semantics and prediction results. This paper proposes a novel influence score (I-score), a greedy search algorithm, called Backward Dropping Algorithm (BDA), and a novel feature engineering technique called the “dagger technique”. First, the paper proposes to use the novel influence score (I-score) to detect and search for the important language semantics in text documents that are useful for making good predictions in text classification tasks. Next, a greedy search algorithm, called the Backward Dropping Algorithm, is proposed to handle long-term dependencies in the dataset. Moreover, the paper proposes a novel engineering technique called the “dagger technique” that fully preserves the relationship between the explanatory variable and the response variable. The proposed techniques can be further generalized into any feed-forward Artificial Neural Networks (ANNs) and Convolutional Neural Networks (CNNs), and any neural network. A real-world application on the Internet Movie Database (IMDB) is used and the proposed methods are applied to improve prediction performance with an 81% error reduction compared to other popular peers if I-score and “dagger technique” are not implemented.

scholarly journals Knowledge-driven graph similarity for text classification

2020 ◽  
Author(s):  
Niloofer Shanavas ◽  
Hui Wang ◽  
Zhiwei Lin ◽  
Glenn Hawe
Keyword(s):  
Text Classification ◽  
Structural Information ◽  
Similarity Measures ◽  
Semantic Knowledge ◽  
Exact Matching ◽  
Text Documents ◽  
Graph Kernel ◽  
Word Similarity ◽  
Graph Similarity ◽  
Automatic Text Classification

AbstractAutomatic text classification using machine learning is significantly affected by the text representation model. The structural information in text is necessary for natural language understanding, which is usually ignored in vector-based representations. In this paper, we present a graph kernel-based text classification framework which utilises the structural information in text effectively through the weighting and enrichment of a graph-based representation. We introduce weighted co-occurrence graphs to represent text documents, which weight the terms and their dependencies based on their relevance to text classification. We propose a novel method to automatically enrich the weighted graphs using semantic knowledge in the form of a word similarity matrix. The similarity between enriched graphs, knowledge-driven graph similarity, is calculated using a graph kernel. The semantic knowledge in the enriched graphs ensures that the graph kernel goes beyond exact matching of terms and patterns to compute the semantic similarity of documents. In the experiments on sentiment classification and topic classification tasks, our knowledge-driven similarity measure significantly outperforms the baseline text similarity measures on five benchmark text classification datasets.

scholarly journals The Unreasonable Effectiveness of the Baseline: Discussing SVMs in Legal Text Classification

2021 ◽  
Author(s):  
Benjamin Clavié ◽  
Marc Alphonsus
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Traditional Approach ◽  
Error Reduction ◽  
Support Vector ◽  
Learning Models ◽  
Legal Text ◽  
Classification Tasks ◽  
Legal Domain

We aim to highlight an interesting trend to contribute to the ongoing debate around advances within legal Natural Language Processing. Recently, the focus for most legal text classification tasks has shifted towards large pre-trained deep learning models such as BERT. In this paper, we show that a more traditional approach based on Support Vector Machine classifiers reaches competitive performance with deep learning models. We also highlight that error reduction obtained by using specialised BERT-based models over baselines is noticeably smaller in the legal domain when compared to general language tasks. We discuss some hypotheses for these results to support future discussions.

scholarly journals A comparative review on deep learning models for text classification

2020 ◽  
Vol 19 (1) ◽  
pp. 325
Author(s):  
Muhammad Zulqarnain ◽  
Rozaida Ghazali ◽  
Yana Mazwin Mohmad Hassim ◽  
Muhammad Rehan
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Question Answering ◽  
Learning Models ◽  
Semantic Classification ◽  
Analysis Question ◽  
Comparative Review ◽  
Classification Tasks

<p>Text classification is a fundamental task in several areas of natural language processing (NLP), including words semantic classification, sentiment analysis, question answering, or dialog management. This paper investigates three basic architectures of deep learning models for the tasks of text classification: Deep Belief Neural (DBN), Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), these three main types of deep learning architectures, are largely explored to handled various classification tasks. DBN have excellent learning capabilities to extracts highly distinguishable features and good for general purpose. CNN have supposed to be better at extracting the position of various related features while RNN is modeling in sequential of long-term dependencies. This paper work shows the systematic comparison of DBN, CNN, and RNN on text classification tasks. Finally, we show the results of deep models by research experiment. The aim of this paper to provides basic guidance about the deep learning models that which models are best for the task of text classification.</p>

scholarly journals IDC: Quantitative Evaluation Benchmark of Interpretation Methods for Deep Text Classification Models

2021 ◽  
Author(s):  
Mohammed Khaleel ◽  
Lei Qi ◽  
Wallapak Tavanapong ◽  
Johnny Wong ◽  
Adisak Sukul ◽  
...  
Keyword(s):  
Quantitative Evaluation ◽  
Language Processing ◽  
Text Classification ◽  
Deep Neural Networks ◽  
Performance Metrics ◽  
Ground Truth ◽  
Decision Making Process ◽  
Research Attention ◽  
Prediction Confidence ◽  
Insight Into

Abstract Recent advances in deep neural networks have achieved outstanding success in natural language processing. Due to the success and the black-box nature of the deep models, interpretation methods that provide insight into the decision-making process of the models have received an influx of research attention. However, there is no quantitative evaluation comparing interpretation methods for text classification other than observing classification accuracy or prediction confidence when important word grams are removed. This is due to the lack of interpretation ground truth. Manual labeling of a large interpretation ground truth is time-consuming. We propose IDC, a new benchmark for quantitative evaluation of I nterpretation methods for D eep text C lassification models. IDC consists of three methods that take existing text classification ground truth and generate three corresponding pseudo-interpretation ground truth datasets. We propose to use interpretation recall, interpretation precision, and Cohen’s kappa inter-agreement as performance metrics. We used the pseudo ground truth datasets and the metrics to evaluate six interpretation methods.

scholarly journals Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks

2018 ◽  
Author(s):  
Renjie Zheng ◽  
Junkun Chen ◽  
Xipeng Qiu
Keyword(s):  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Representation Learning ◽  
Training Data ◽  
Specific Information ◽  
Distributed Representation ◽  
Source Codes ◽  
Multiple Tasks ◽  
Classification Tasks

Distributed representation plays an important role in deep learning based natural language processing. However, the representation of a sentence often varies in different tasks, which is usually learned from scratch and suffers from the limited amounts of training data. In this paper, we claim that a good sentence representation should be invariant and can benefit the various subsequent tasks. To achieve this purpose, we propose a new scheme of information sharing for multi-task learning. More specifically, all tasks share the same sentence representation and each task can select the task-specific information from the shared sentence representation with attention mechanisms. The query vector of each task's attention could be either static parameters or generated dynamically. We conduct extensive experiments on 16 different text classification tasks, which demonstrate the benefits of our architecture. Source codes of this paper are available on Github.

scholarly journals Distance Variance Score: An Efficient Feature Selection Method in Text Classification

2015 ◽  
Vol 2015 ◽  
pp. 1-10 ◽  
Author(s):  
Heyong Wang ◽  
Ming Hong
Keyword(s):  
Feature Selection ◽  
Text Mining ◽  
Text Classification ◽  
Web Applications ◽  
Rapid Development ◽  
Feature Selection Method ◽  
Selection Method ◽  
Text Documents ◽  
Unsupervised Feature Selection ◽  
Classification Feature

With the rapid development of web applications such as social network, a large amount of electric text data is accumulated and available on the Internet, which causes increasing interests in text mining. Text classification is one of the most important subfields of text mining. In fact, text documents are often represented as a high-dimensional sparse document term matrix (DTM) before classification. Feature selection is essential and vital for text classification due to high dimensionality and sparsity of DTM. An efficient feature selection method is capable of both reducing dimensions of DTM and selecting discriminative features for text classification. Laplacian Score (LS) is one of the unsupervised feature selection methods and it has been successfully used in areas such as face recognition. However, LS is unable to select discriminative features for text classification and to effectively reduce the sparsity of DTM. To improve it, this paper proposes an unsupervised feature selection method named Distance Variance Score (DVS). DVS uses feature distance contribution (a ratio) to rank the importance of features for text documents so as to select discriminative features. Experimental results indicate that DVS is able to select discriminative features and reduce the sparsity of DTM. Thus, it is much more efficient than LS.

scholarly journals Inspecting Hybrid Data Mining Approaches in Decision Support Systems for Humanities Texts Criticism

2021 ◽  
pp. 4101-4109
Author(s):  
Baraa Hasan Hadi ◽  
Tareef Kamil Mustafa
Keyword(s):  
Data Mining ◽  
Decision Support ◽  
Text Mining ◽  
Language Processing ◽  
Hybrid Approach ◽  
Decision Support Techniques ◽  
Relevant Information ◽  
Delta Method ◽  
Technological Advancement ◽  
Text Documents

The majority of systems dealing with natural language processing (NLP) and artificial intelligence (AI) can assist in making automated and automatically-supported decisions. However, these systems may face challenges and difficulties or find it confusing to identify the required information (characterization) for eliciting a decision by extracting or summarizing relevant information from large text documents or colossal content.   When obtaining these documents online, for instance from social networking or social media, these sites undergo a remarkable increase in the textual content. The main objective of the present study is to conduct a survey and show the latest developments about the implementation of text-mining techniques in humanities when summarizing and eliciting automated decisions. This process relies on technological advancement and considers (1) the automated-decision support-techniques commonly used in humanities, (2) the performance evolution and the use of the stylometric approach in text-mining, and (3) the comparisons of the results of chunking text by using different attributes in Burrows' Delta method. This study also provides an overview of the efficiency of applying some selected data-mining (DM) methods with various text-mining techniques to support the critics' decision in artistry ‒ one field of humanities. The automatic choice of criticism in this field was supported by a hybrid approach to these procedures.

Sign in / Sign up

Export Citation Format

Share Document