scholarly journals Automatic Taxonomy Classification by Pretrained Language Model

Electronics ◽  
2021 ◽  
Vol 10 (21) ◽  
pp. 2656
Author(s):  
Ayato Kuwana ◽  
Atsushi Oba ◽  
Ranto Sawai ◽  
Incheon Paik
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Question Answering ◽  
Information Science ◽  
Language Models ◽  
Real Field ◽  
Future Research ◽  
Initial Attempt ◽  
Online Data ◽  
Ontology Generation

In recent years, automatic ontology generation has received significant attention in information science as a means of systemizing vast amounts of online data. As our initial attempt of ontology generation with a neural network, we proposed a recurrent neural network-based method. However, updating the architecture is possible because of the development in natural language processing (NLP). By contrast, the transfer learning of language models trained by a large, unlabeled corpus has yielded a breakthrough in NLP. Inspired by these achievements, we propose a novel workflow for ontology generation comprising two-stage learning. Our results showed that our best method improved accuracy by over 12.5%. As an application example, we applied our model to the Stanford Question Answering Dataset to show ontology generation in a real field. The results showed that our model can generate a good ontology, with some exceptions in the real field, indicating future research directions to improve the quality.

scholarly journals Arabic Fake News Detection: Comparative Study of Neural Networks and Transformer-Based Approaches

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Maha Al-Yahya ◽  
Hend Al-Khalifa ◽  
Heyam Al-Baity ◽  
Duaa AlSaeed ◽  
Amr Essam
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Comparative Study ◽  
Language Processing ◽  
Language Models ◽  
News Article ◽  
Future Research ◽  
Fake News ◽  
The Neural Network ◽  
The Difference

Fake news detection (FND) involves predicting the likelihood that a particular news article (news report, editorial, expose, etc.) is intentionally deceptive. Arabic FND started to receive more attention in the last decade, and many detection approaches demonstrated some ability to detect fake news on multiple datasets. However, most existing approaches do not consider recent advances in natural language processing, i.e., the use of neural networks and transformers. This paper presents a comprehensive comparative study of neural network and transformer-based language models used for Arabic FND. We examine the use of neural networks and transformer-based language models for Arabic FND and show their performance compared to each other. We also conduct an extensive analysis of the possible reasons for the difference in performance results obtained by different approaches. The results demonstrate that transformer-based models outperform the neural network-based solutions, which led to an increase in the F1 score from 0.83 (best neural network-based model, GRU) to 0.95 (best transformer-based model, QARiB), and it boosted the accuracy by 16% compared to the best in neural network-based solutions. Finally, we highlight the main gaps in Arabic FND research and suggest future research directions.

scholarly journals Efficient Embedded Decoding of Neural Network Language Models in a Machine Translation System

2018 ◽  
Vol 28 (09) ◽  
pp. 1850007
Author(s):  
Francisco Zamora-Martinez ◽  
Maria Jose Castro-Bleda
Keyword(s):  
Neural Network ◽  
Machine Translation ◽  
Language Processing ◽  
Traditional Approach ◽  
Computational Cost ◽  
Integrated Approach ◽  
Language Models ◽  
Translation System ◽  
Neural Net ◽  
Network Language

Neural Network Language Models (NNLMs) are a successful approach to Natural Language Processing tasks, such as Machine Translation. We introduce in this work a Statistical Machine Translation (SMT) system which fully integrates NNLMs in the decoding stage, breaking the traditional approach based on [Formula: see text]-best list rescoring. The neural net models (both language models (LMs) and translation models) are fully coupled in the decoding stage, allowing to more strongly influence the translation quality. Computational issues were solved by using a novel idea based on memorization and smoothing of the softmax constants to avoid their computation, which introduces a trade-off between LM quality and computational cost. These ideas were studied in a machine translation task with different combinations of neural networks used both as translation models and as target LMs, comparing phrase-based and [Formula: see text]-gram-based systems, showing that the integrated approach seems more promising for [Formula: see text]-gram-based systems, even with nonfull-quality NNLMs.

scholarly journals Evaluating Commonsense in Pre-Trained Language Models

2020 ◽  
Vol 34 (05) ◽  
pp. 9733-9740 ◽  
Author(s):  
Xuhui Zhou ◽  
Yue Zhang ◽  
Leyang Cui ◽  
Dandan Huang
Keyword(s):  
Reading Comprehension ◽  
Question Answering ◽  
Deep Level ◽  
Language Models ◽  
Future Research ◽  
Correct Prediction ◽  
Test Cases ◽  
Word Sense ◽  
Training Set ◽  
Text Data

Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension. There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks. However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension. We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses. We additionally find that current models do poorly on tasks require more necessary inference steps. Finally, we test the robustness of models by making dual test cases, which are correlated so that the correct prediction of one sample should lead to correct prediction of the other. Interestingly, the models show confusion on these test cases, which suggests that they learn commonsense at the surface rather than the deep level. We release a test set, named CATs publicly, for future research.

scholarly journals Analyzing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasets

2021 ◽  
Vol 2021 ◽  
pp. 1-17
Author(s):  
Changchang Zeng ◽  
Shaobo Li
Keyword(s):  
Reading Comprehension ◽  
Language Processing ◽  
Question Answering ◽  
Multiple Choice ◽  
Length Distribution ◽  
Research Field ◽  
Evaluation Framework ◽  
Language Models ◽  
Training Objective ◽  
Machine Reading

Machine reading comprehension (MRC) is a challenging natural language processing (NLP) task. It has a wide application potential in the fields of question answering robots, human-computer interactions in mobile virtual reality systems, etc. Recently, the emergence of pretrained models (PTMs) has brought this research field into a new era, in which the training objective plays a key role. The masked language model (MLM) is a self-supervised training objective widely used in various PTMs. With the development of training objectives, many variants of MLM have been proposed, such as whole word masking, entity masking, phrase masking, and span masking. In different MLMs, the length of the masked tokens is different. Similarly, in different machine reading comprehension tasks, the length of the answer is also different, and the answer is often a word, phrase, or sentence. Thus, in MRC tasks with different answer lengths, whether the length of MLM is related to performance is a question worth studying. If this hypothesis is true, it can guide us on how to pretrain the MLM with a relatively suitable mask length distribution for MRC tasks. In this paper, we try to uncover how much of MLM’s success in the machine reading comprehension tasks comes from the correlation between masking length distribution and answer length in the MRC dataset. In order to address this issue, herein, (1) we propose four MRC tasks with different answer length distributions, namely, the short span extraction task, long span extraction task, short multiple-choice cloze task, and long multiple-choice cloze task; (2) four Chinese MRC datasets are created for these tasks; (3) we also have pretrained four masked language models according to the answer length distributions of these datasets; and (4) ablation experiments are conducted on the datasets to verify our hypothesis. The experimental results demonstrate that our hypothesis is true. On four different machine reading comprehension datasets, the performance of the model with correlation length distribution surpasses the model without correlation.

Bilingual Mental Lexicon and Collocational Processing

2018 ◽  
pp. 221-243
Author(s):  
Hakan Cangır
Keyword(s):  
Language Processing ◽  
Cognitive Linguistics ◽  
Mental Lexicon ◽  
Research Literature ◽  
Language Models ◽  
Future Research ◽  
Formulaic Language ◽  
Lexical Activation

The chapter starts with a definition and models of mental dictionary. It then builds on the bilingual lexical activation models and goes on to discuss formulaic language (collocations in particular). After explaining the basics of formulaic language processing, the author attempts to address the issue of lexical and collocational priming theory by Hoey, which has its roots in cognitive linguistics and usage-based language models. Last but not least, some suggestions for future research are provided in an attempt to address the needs of the lexical research literature in the Turkish setting.

An Improved Cross-Domain Sentiment Analysis Based on a Semi-Supervised Convolutional Neural Network

2022 ◽  
pp. 155-170
Author(s):  
Lap-Kei Lee ◽  
Kwok Tai Chui ◽  
Jingjing Wang ◽  
Yin-Chun Fung ◽  
Zhanhui Tan
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
Sentiment Analysis ◽  
Language Processing ◽  
Research Work ◽  
Latent Semantic Indexing ◽  
Future Research ◽  
Training Time ◽  
Cross Domain ◽  
Document Frequency

The dependence on Internet in our daily life is ever-growing, which provides opportunity to discover valuable and subjective information using advanced techniques such as natural language processing and artificial intelligence. In this chapter, the research focus is a convolutional neural network for three-class (positive, neutral, and negative) cross-domain sentiment analysis. The model is enhanced in two-fold. First, a similarity label method facilitates the management between the source and target domains to generate more labelled data. Second, term frequency-inverse document frequency (TF-IDF) and latent semantic indexing (LSI) are employed to compute the similarity between source and target domains. Performance evaluation is conducted using three datasets, beauty reviews, toys reviews, and phone reviews. The proposed method enhances the accuracy by 4.3-7.6% and reduces the training time by 50%. The limitations of the research work have been discussed, which serve as the rationales of future research directions.

scholarly journals A comparative review on deep learning models for text classification

2020 ◽  
Vol 19 (1) ◽  
pp. 325
Author(s):  
Muhammad Zulqarnain ◽  
Rozaida Ghazali ◽  
Yana Mazwin Mohmad Hassim ◽  
Muhammad Rehan
Keyword(s):  
Neural Network ◽  
Deep Learning ◽  
Language Processing ◽  
Text Classification ◽  
Question Answering ◽  
Learning Models ◽  
Semantic Classification ◽  
Analysis Question ◽  
Comparative Review ◽  
Classification Tasks

<p>Text classification is a fundamental task in several areas of natural language processing (NLP), including words semantic classification, sentiment analysis, question answering, or dialog management. This paper investigates three basic architectures of deep learning models for the tasks of text classification: Deep Belief Neural (DBN), Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), these three main types of deep learning architectures, are largely explored to handled various classification tasks. DBN have excellent learning capabilities to extracts highly distinguishable features and good for general purpose. CNN have supposed to be better at extracting the position of various related features while RNN is modeling in sequential of long-term dependencies. This paper work shows the systematic comparison of DBN, CNN, and RNN on text classification tasks. Finally, we show the results of deep models by research experiment. The aim of this paper to provides basic guidance about the deep learning models that which models are best for the task of text classification.</p>

Bilingual Mental Lexicon and Collocational Processing

2022 ◽  
pp. 706-722
Author(s):  
Hakan Cangır
Keyword(s):  
Language Processing ◽  
Cognitive Linguistics ◽  
Mental Lexicon ◽  
Research Literature ◽  
Language Models ◽  
Future Research ◽  
Formulaic Language ◽  
Lexical Activation

The chapter starts with a definition and models of mental dictionary. It then builds on the bilingual lexical activation models and goes on to discuss formulaic language (collocations in particular). After explaining the basics of formulaic language processing, the author attempts to address the issue of lexical and collocational priming theory by Hoey, which has its roots in cognitive linguistics and usage-based language models. Last but not least, some suggestions for future research are provided in an attempt to address the needs of the lexical research literature in the Turkish setting.

scholarly journals From ‘F’ to ‘A’ on the N.Y. Regents Science Exams: An Overview of the Aristo Project

AI Magazine ◽  
2020 ◽  
Vol 41 (4) ◽  
pp. 39-53
Author(s):  
Peter Clark ◽  
Oren Etzioni ◽  
Tushar Khot ◽  
Daniel Khashabi ◽  
Bhavana Mishra ◽  
...  
Keyword(s):  
New York ◽  
Language Processing ◽  
Question Answering ◽  
Multiple Choice ◽  
Language Models ◽  
General Question ◽  
8Th Grade ◽  
The Rich ◽  
Full Solution ◽  
Standardized Exams

AI has achieved remarkable mastery over games such as Chess, Go, and Poker, and even Jeopardy!, but the rich variety of standardized exams has remained a landmark challenge. Even as recently as 2016, the best AI system could achieve merely 59.3 percent on an 8th grade science exam. This article reports success on the Grade 8 New York Regents Science Exam, where for the first time a system scores more than 90 percent on the exam’s nondiagram, multiple choice (NDMC) questions. In addition, our Aristo system, building upon the success of recent language models, exceeded 83 percent on the corresponding Grade 12 Science Exam NDMC questions. The results, on unseen test questions, are robust across different test years and different variations of this kind of test. They demonstrate that modern natural language processing methods can result in mastery on this task. While not a full solution to general question-answering (the questions are limited to 8th grade multiple-choice science) it represents a significant milestone for the field.

scholarly journals Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge

2022 ◽  
Vol 40 (1) ◽  
pp. 1-33
Author(s):  
Yang Deng ◽  
Yuexiang Xie ◽  
Yaliang Li ◽  
Min Yang ◽  
Wai Lam ◽  
...  
Keyword(s):  
Neural Network ◽  
Language Processing ◽  
Question Answering ◽  
Representation Learning ◽  
Attention Mechanism ◽  
Convolutional Network ◽  
Structure Information ◽  
Dialog Systems ◽  
External Knowledge ◽  
Knowledge Based

Answer selection, which is involved in many natural language processing applications, such as dialog systems and question answering (QA), is an important yet challenging task in practice, since conventional methods typically suffer from the issues of ignoring diverse real-world background knowledge. In this article, we extensively investigate approaches to enhancing the answer selection model with external knowledge from knowledge graph (KG). First, we present a context-knowledge interaction learning framework, Knowledge-aware Neural Network, which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information. Then, we develop two kinds of knowledge-aware attention mechanism to summarize both the context-based and knowledge-based interactions between questions and answers. To handle the diversity and complexity of KG information, we further propose a Contextualized Knowledge-aware Attentive Neural Network, which improves the knowledge representation learning with structure information via a customized Graph Convolutional Network and comprehensively learns context-based and knowledge-based sentence representation via the multi-view knowledge-aware attention mechanism. We evaluate our method on four widely used benchmark QA datasets, including WikiQA, TREC QA, InsuranceQA, and Yahoo QA. Results verify the benefits of incorporating external knowledge from KG and show the robust superiority and extensive applicability of our method.

Sign in / Sign up

Export Citation Format

Share Document