scholarly journals Evaluating Computational Language Models with Scaling Properties of Natural Language

2019 ◽  
Vol 45 (3) ◽  
pp. 481-513
Author(s):  
Shuntaro Takahashi ◽  
Kumiko Tanaka-Ishii
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Recurrent Neural Networks ◽  
Long Memory ◽  
Computational Models ◽  
Language Models ◽  
Generative Adversarial Networks ◽  
Scaling Properties ◽  
Gating Mechanism ◽  
Taylor’S Law

In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf’s law, Heaps’ law, Ebeling’s method, Taylor’s law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test n-gram language models, a probabilistic context-free grammar, language models based on Simon/Pitman-Yor processes, neural language models, and generative adversarial networks for text generation. Our analysis reveals that language models based on recurrent neural networks with a gating mechanism (i.e., long short-term memory; a gated recurrent unit; and quasi-recurrent neural networks) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor’s law is a good indicator of model quality.

Locally Recurrent Neural Networks and Their Applications

2010 ◽  
pp. 195-222
Author(s):  
Todor D. Ganchev
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Computational Models ◽  
Machine Learning Techniques ◽  
Spatial Correlations ◽  
Temporal Correlations ◽  
Learning Techniques ◽  
Locally Recurrent ◽  
The One ◽  
And Control

In this chapter we review various computational models of locally recurrent neurons and deliberate the architecture of some archetypal locally recurrent neural networks (LRNNs) that are based on them. Generalizations of these structures are discussed as well. Furthermore, we point at a number of realworld applications of LRNNs that have been reported in past and recent publications. These applications involve classification or prediction of temporal sequences, discovering and modeling of spatial and temporal correlations, process identification and control, etc. Validation experiments reported in these developments provide evidence that locally recurrent architectures are capable of identifying and exploiting temporal and spatial correlations (i.e., the context in which events occur), which is the main reason for their advantageous performance when compared with the one of their non-recurrent counterparts or other reasonable machine learning techniques.

scholarly journals Translating Videos to Natural Language Using Deep Recurrent Neural Networks

2015 ◽  
Author(s):  
Subhashini Venugopalan ◽  
Huijuan Xu ◽  
Jeff Donahue ◽  
Marcus Rohrbach ◽  
Raymond Mooney ◽  
...  
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Recurrent Neural Networks

SQL Generation from Natural Language Using Supervised Learning and Recurrent Neural Networks

2020 ◽  
pp. 175-183
Author(s):  
Youssef Mellah ◽  
El Hassane Ettifouri ◽  
Abdelkader Rhouati ◽  
Walid Dahhane ◽  
Toumi Bouchentouf ◽  
...  
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Supervised Learning ◽  
Recurrent Neural Networks

scholarly journals Deep learning in Emergency Medicine: Recent contributions and methodological challenges

2020 ◽  
Vol 16 (1) ◽  
Author(s):  
Francesco Faita
Keyword(s):  
Artificial Intelligence ◽  
Neural Networks ◽  
Emergency Medicine ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Recurrent Neural Networks ◽  
Computational Models ◽  
Clinical Usefulness ◽  
Human Knowledge ◽  
Deep Convolutional Neural Networks

In the last few years, artificial intelligence (AI) technology has grown dramatically impacting several fields of human knowledge and medicine in particular. Among other approaches, deep learning, which is a subset of AI based on specific computational models, such as deep convolutional neural networks and recurrent neural networks, has shown exceptional performance in images and signals processing. Accordingly, emergency medicine will benefit from the adoption of this technology. However, a particular attention should be devoted to the review of these papers in order to exclude overoptimistic results from clinically transferable ones. We presented a group of studies recently published on PubMed and selected by keywords ‘deep learning emergency medicine’ and ‘artificial intelligence emergency medicine’ with the aim of highlighting their methodological strengths and weaknesses, as well as their clinical usefulness.

scholarly journals Sentence embeddings in NLI with iterative refinement encoders

2019 ◽  
Vol 25 (4) ◽  
pp. 467-482 ◽  
Author(s):  
Aarne Talman ◽  
Anssi Yli-Jyrä ◽  
Jörg Tiedemann
Keyword(s):  
Neural Networks ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
State Of The Art ◽  
Iterative Refinement ◽  
Learning Tasks ◽  
Sentence Level ◽  
Refinement Strategy

AbstractSentence-level representations are necessary for various natural language processing tasks. Recurrent neural networks have proven to be very effective in learning distributed representations and can be trained efficiently on natural language inference tasks. We build on top of one such model and propose a hierarchy of bidirectional LSTM and max pooling layers that implements an iterative refinement strategy and yields state of the art results on the SciTail dataset as well as strong results for Stanford Natural Language Inference and Multi-Genre Natural Language Inference. We can show that the sentence embeddings learned in this way can be utilized in a wide variety of transfer learning tasks, outperforming InferSent on 7 out of 10 and SkipThought on 8 out of 9 SentEval sentence embedding evaluation tasks. Furthermore, our model beats the InferSent model in 8 out of 10 recently published SentEval probing tasks designed to evaluate sentence embeddings’ ability to capture some of the important linguistic properties of sentences.

Sign in / Sign up

Export Citation Format

Share Document