Towards recurrent neural networks language models with linguistic and contextual features

2012 ◽  
Author(s):  
Yangyang Shi ◽  
Pascal Wiggers ◽  
Catholijn M. Jonker
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Language Models ◽  
Contextual Features

Combining recurrent neural networks and factored language models during decoding of code-Switching speech

2014 ◽  
Author(s):  
Heike Adel ◽  
Dominic Telaar ◽  
Ngoc Thang Vu ◽  
Katrin Kirchhoff ◽  
Tanja Schultz
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Code Switching ◽  
Language Models

Towards an Embedding-Based Approach for the Geolocation of Texts and Users on Social Networks

2021 ◽  
pp. 206-234
Author(s):  
Sarra Hasni
Keyword(s):  
Neural Networks ◽  
Social Networks ◽  
Spatial Analysis ◽  
Recurrent Neural Networks ◽  
Contextual Information ◽  
Language Models ◽  
New Approach ◽  
Textual Data ◽  
Word Forms ◽  
Statistical Language Models

The geolocation task of textual data shared on social networks like Twitter attracts a progressive attention. Since those data are supported by advanced geographic information systems for multipurpose spatial analysis, new trends to extend the paradigm of geolocated data become more emergent. Differently from statistical language models that are widely adopted in prior works, the authors propose a new approach that is adopted to the geolocation of both tweets and users through the application of embedding models. The authors boost the geolocation strategy with a sequential modelling using recurrent neural networks to delimit the importance of words in tweets with respect to contextual information. They evaluate the power of this strategy in order to determine locations of unstructured texts that reflect unlimited user's writing styles. Especially, the authors demonstrate that semantic proprieties and word forms can be effective to geolocate texts without specifying local words or topics' descriptions per region.

scholarly journals Representation of Linguistic Form and Function in Recurrent Neural Networks

2017 ◽  
Vol 43 (4) ◽  
pp. 761-780 ◽  
Author(s):  
Ákos Kádár ◽  
Grzegorz Chrupała ◽  
Afra Alishahi
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Network Architecture ◽  
Language Model ◽  
Visual Pathway ◽  
Recurrent Network ◽  
Point Of View ◽  
Language Models ◽  
Grammatical Function ◽  
Form And Function

We present novel methods for analyzing the activation patterns of recurrent neural networks from a linguistic point of view and explore the types of linguistic structure they learn. As a case study, we use a standard standalone language model, and a multi-task gated recurrent network architecture consisting of two parallel pathways with shared word embeddings: The Visual pathway is trained on predicting the representations of the visual scene corresponding to an input sentence, and the Textual pathway is trained to predict the next word in the same sentence. We propose a method for estimating the amount of contribution of individual tokens in the input to the final prediction of the networks. Using this method, we show that the Visual pathway pays selective attention to lexical categories and grammatical functions that carry semantic information, and learns to treat word types differently depending on their grammatical function and their position in the sequential structure of the sentence. In contrast, the language models are comparatively more sensitive to words with a syntactic function. Further analysis of the most informative n-gram contexts for each model shows that in comparison with the Visual pathway, the language models react more strongly to abstract contexts that represent syntactic constructions.

scholarly journals Tabula Nearly Rasa: Probing the Linguistic Knowledge of Character-level Neural Language Models Trained on Unsegmented Text

2019 ◽  
Vol 7 ◽  
pp. 467-484 ◽  
Author(s):  
Michael Hahn ◽  
Marco Baroni
Keyword(s):  
Neural Networks ◽  
Language Learning ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
Input Data ◽  
Language Models ◽  
Linguistic Knowledge ◽  
Word Level ◽  
Linguistic Unit ◽  
Word Boundaries

Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks. This has renewed interest in whether these generic sequence processing devices are inducing genuine linguistic knowledge. Nearly all current analytical studies, however, initialize the RNNs with a vocabulary of known words, and feed them tokenized input during training. We present a multi-lingual study of the linguistic knowledge encoded in RNNs trained as character-level language models, on input data with word boundaries removed. These networks face a tougher and more cognitively realistic task, having to discover any useful linguistic unit from scratch based on input statistics. The results show that our “near tabula rasa” RNNs are mostly able to solve morphological, syntactic and semantic tasks that intuitively presuppose word-level knowledge, and indeed they learned, to some extent, to track word boundaries. Our study opens the door to speculations about the necessity of an explicit, rigid word lexicon in language learning and usage.

Comparing approaches to convert recurrent neural networks into backoff language models for efficient decoding

2014 ◽  
Author(s):  
Heike Adel ◽  
Katrin Kirchhoff ◽  
Ngoc Thang Vu ◽  
Dominic Telaar ◽  
Tanja Schultz
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Language Models ◽  
Efficient Decoding

scholarly journals Evaluating Computational Language Models with Scaling Properties of Natural Language

2019 ◽  
Vol 45 (3) ◽  
pp. 481-513
Author(s):  
Shuntaro Takahashi ◽  
Kumiko Tanaka-Ishii
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Recurrent Neural Networks ◽  
Long Memory ◽  
Computational Models ◽  
Language Models ◽  
Generative Adversarial Networks ◽  
Scaling Properties ◽  
Gating Mechanism ◽  
Taylor’S Law

In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf’s law, Heaps’ law, Ebeling’s method, Taylor’s law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test n-gram language models, a probabilistic context-free grammar, language models based on Simon/Pitman-Yor processes, neural language models, and generative adversarial networks for text generation. Our analysis reveals that language models based on recurrent neural networks with a gating mechanism (i.e., long short-term memory; a gated recurrent unit; and quasi-recurrent neural networks) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor’s law is a good indicator of model quality.

Sign in / Sign up

Export Citation Format

Share Document