scholarly journals Natural language grammatical inference with recurrent neural networks

2000 ◽  
Vol 12 (1) ◽  
pp. 126-140 ◽  
Author(s):  
S. Lawrence ◽  
C.L. Giles ◽  
S. Fong
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Recurrent Neural Networks ◽  
Grammatical Inference

EXPERIMENTAL COMPARISON OF THE EFFECT OF ORDER IN RECURRENT NEURAL NETWORKS

1993 ◽  
Vol 07 (04) ◽  
pp. 849-872 ◽  
Author(s):  
CLIFFORD B. MILLER ◽  
C. LEE GILES
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Internal State ◽  
Second Order ◽  
Convergence Time ◽  
Experimental Comparison ◽  
Grammatical Inference ◽  
Neural Net ◽  
First Order ◽  
Finite State

There has been much interest in increasing the computational power of neural networks. In addition there has been much interest in “designing” neural networks better suited to particular problems. Increasing the “order” of the connectivity of a neural network permits both. Though order has played a significant role in feedforward neural networks, its role in dynamically driven recurrent networks is still being understood. This work explores the effect of order in learning grammars. We present an experimental comparison of first order and second order recurrent neural networks, as applied to the task of grammatical inference. We show that for the small grammars studied these two neural net architectures have comparable learning and generalization power, and that both are reasonably capable of extracting the correct finite state automata for the language in question. However, for a larger randomly-generated ten-state grammar, second order networks significantly outperformed the first order networks, both in convergence time and generalization capability. We show that these networks learn faster the more neurons they have (our experiments used up to 10 hidden neurons), but that the solutions found by smaller networks are usually of better quality (in terms of generalization performance after training). Second order nets have the advantage that they converge more quickly to a solution and can find it more reliably than first order nets, but that the second order solutions tend to be of poorer quality than those of the first order if both architectures are trained to the same error tolerance. Despite this, second order nets can more successfully extract finite state machines using heuristic clustering techniques applied to the internal state representations. We speculate that this may be due to restrictions on the ability of first order architecture to fully make use of its internal state representation power and that this may have implications for the performance of the two architectures when scaled up to larger problems.

scholarly journals Discrete recurrent neural networks for grammatical inference

1994 ◽  
Vol 5 (2) ◽  
pp. 320-330 ◽  
Author(s):  
Zheng Zeng ◽  
R.M. Goodman ◽  
P. Smyth
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Grammatical Inference

scholarly journals Translating Videos to Natural Language Using Deep Recurrent Neural Networks

2015 ◽  
Author(s):  
Subhashini Venugopalan ◽  
Huijuan Xu ◽  
Jeff Donahue ◽  
Marcus Rohrbach ◽  
Raymond Mooney ◽  
...  
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Recurrent Neural Networks

SQL Generation from Natural Language Using Supervised Learning and Recurrent Neural Networks

2020 ◽  
pp. 175-183
Author(s):  
Youssef Mellah ◽  
El Hassane Ettifouri ◽  
Abdelkader Rhouati ◽  
Walid Dahhane ◽  
Toumi Bouchentouf ◽  
...  
Keyword(s):  
Neural Networks ◽  
Natural Language ◽  
Supervised Learning ◽  
Recurrent Neural Networks

scholarly journals Sentence embeddings in NLI with iterative refinement encoders

2019 ◽  
Vol 25 (4) ◽  
pp. 467-482 ◽  
Author(s):  
Aarne Talman ◽  
Anssi Yli-Jyrä ◽  
Jörg Tiedemann
Keyword(s):  
Neural Networks ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Recurrent Neural Networks ◽  
State Of The Art ◽  
Iterative Refinement ◽  
Learning Tasks ◽  
Sentence Level ◽  
Refinement Strategy

AbstractSentence-level representations are necessary for various natural language processing tasks. Recurrent neural networks have proven to be very effective in learning distributed representations and can be trained efficiently on natural language inference tasks. We build on top of one such model and propose a hierarchy of bidirectional LSTM and max pooling layers that implements an iterative refinement strategy and yields state of the art results on the SciTail dataset as well as strong results for Stanford Natural Language Inference and Multi-Genre Natural Language Inference. We can show that the sentence embeddings learned in this way can be utilized in a wide variety of transfer learning tasks, outperforming InferSent on 7 out of 10 and SkipThought on 8 out of 9 SentEval sentence embedding evaluation tasks. Furthermore, our model beats the InferSent model in 8 out of 10 recently published SentEval probing tasks designed to evaluate sentence embeddings’ ability to capture some of the important linguistic properties of sentences.

Sign in / Sign up

Export Citation Format

Share Document