scholarly journals An Entropy Metric for Regular Grammar Classification and Learning with Recurrent Neural Networks

Entropy ◽  
2021 ◽  
Vol 23 (1) ◽  
pp. 127
Author(s):  
Kaixuan Zhang ◽  
Qinglong Wang ◽  
C. Lee Giles
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Formal Language ◽  
Language Theory ◽  
Order Information ◽  
Concentric Ring ◽  
Original Order ◽  
Internal Structures ◽  
Regular Grammar ◽  
Learning Research

Recently, there has been a resurgence of formal language theory in deep learning research. However, most research focused on the more practical problems of attempting to represent symbolic knowledge by machine learning. In contrast, there has been limited research on exploring the fundamental connection between them. To obtain a better understanding of the internal structures of regular grammars and their corresponding complexity, we focus on categorizing regular grammars by using both theoretical analysis and empirical evidence. Specifically, motivated by the concentric ring representation, we relaxed the original order information and introduced an entropy metric for describing the complexity of different regular grammars. Based on the entropy metric, we categorized regular grammars into three disjoint subclasses: the polynomial, exponential and proportional classes. In addition, several classification theorems are provided for different representations of regular grammars. Our analysis was validated by examining the process of learning grammars with multiple recurrent neural networks. Our results show that as expected more complex grammars are generally more difficult to learn.

scholarly journals Modelling a subregular bias in phonological learning with Recurrent Neural Networks

2021 ◽  
Vol 9 (1) ◽  
Author(s):  
Brandon Prickett
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Formal Language ◽  
Formal Language Theory ◽  
Language Theory ◽  
Phonological Learning

A number of experiments have demonstrated what seems to be a bias in human phonological learning for patterns that are simpler according to Formal Language Theory (Finley and Badecker 2008; Lai 2015; Avcu 2018). This paper demonstrates that a sequence-to-sequence neural network (Sutskever et al. 2014), which has no such restriction explicitly built into its architecture, can successfully capture this bias. These results suggest that a bias for patterns that are simpler according to Formal Language Theory may not need to be explicitly incorporated into models of phonological learning.

scholarly journals Static Watson-Crick regular grammar

2018 ◽  
Vol 14 ◽  
pp. 457-462
Author(s):  
Aqilahfarhana Abdul Rahman ◽  
Wan Heng Fong ◽  
Nor Haniza Sarmin ◽  
Sherzod Turaev ◽  
Nurul Liyana Mohamad Zulkufli
Keyword(s):  
Dna Computing ◽  
Formal Language ◽  
Computational Models ◽  
Formal Language Theory ◽  
Language Theory ◽  
Chomsky Hierarchy ◽  
New Variant ◽  
Sticker System ◽  
Dna Strands ◽  
Regular Grammar

DNA computing, or more generally, molecular computing, is a recent development at the interface of computer science and molecular biology. In DNA computing, many computational models have been proposed in the framework of formal language theory and automata such as Watson-Crick grammars and sticker systems. A Watson-Crick grammar is a grammar model that generates double stranded strings, whereas a sticker system is a DNA computing model of the ligation and annealing operations over DNA strands using the Watson-Crick complementarity to form a complete double stranded DNA sequence. Most of the proposed DNA computing models make use of this concept, including the Watson-Crick grammars and sticker systems. Watson-Crick grammars and their variants can be explored using formal language theory which allows the development of new concepts of Watson-Crick grammars.  In this research, a new variant of Watson-Crick grammar called a static Watson-Crick regular grammar is introduced as an analytical counterpart of sticker systems. The computation of a sticker system starts from a given set of incomplete double stranded sequence to form a complete double stranded sequence. Here, a static Watson-Crick regular grammar differs from a dynamic Watson-Crick regular grammar in generating double stranded strings: the latter grammar produces each strand string “independently” and only check for the Watson-Crick complementarity of a generated complete double stranded string at the end, while the former grammar generates both strand strings “dependently”, i.e., checking for the Watson-Crick complementarity for each complete substring. In this paper, computational properties of static Watson-Crick regular grammars are investigated to correlate with the Chomsky hierarchy and hierarchy of the families of dynamic Watson-Crick regular languages. The relationship between families of languages generated by static Watson-Crick regular grammars with several variants of sticker systems, Watson-Crick regular grammars and Chomsky grammars are presented by showing the hierarchy.

Levenshtein Augmentation Improves Performance of SMILES Based Deep-Learning Synthesis Prediction

2020 ◽  
Author(s):  
Dean Sumner ◽  
Jiazhen He ◽  
Amol Thakkar ◽  
Ola Engkvist ◽  
Esben Jannik Bjerrum
Keyword(s):  
Neural Networks ◽  
Pattern Recognition ◽  
Deep Learning ◽  
Recurrent Neural Networks ◽  
Data Augmentation ◽  
State Of The Art ◽  
Sequence Similarity ◽  
Learning Models ◽  
Underlying Network

<p>SMILES randomization, a form of data augmentation, has previously been shown to increase the performance of deep learning models compared to non-augmented baselines. Here, we propose a novel data augmentation method we call “Levenshtein augmentation” which considers local SMILES sub-sequence similarity between reactants and their respective products when creating training pairs. The performance of Levenshtein augmentation was tested using two state of the art models - transformer and sequence-to-sequence based recurrent neural networks with attention. Levenshtein augmentation demonstrated an increase performance over non-augmented, and conventionally SMILES randomization augmented data when used for training of baseline models. Furthermore, Levenshtein augmentation seemingly results in what we define as <i>attentional gain </i>– an enhancement in the pattern recognition capabilities of the underlying network to molecular motifs.</p>

LatticeRnn: Recurrent Neural Networks Over Lattices

2016 ◽  
Author(s):  
Faisal Ladhak ◽  
Ankur Gandhe ◽  
Markus Dreyer ◽  
Lambert Mathias ◽  
Ariya Rastrow ◽  
...  
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks
Sign in / Sign up

Export Citation Format

Share Document