A Dual Simple Recurrent Network Model for Chunking and Abstract Processes in Sequence Learning

Although many studies have provided evidence that abstract knowledge can be acquired in artificial grammar learning, it remains unclear how abstract knowledge can be attained in sequence learning. To address this issue, we proposed a dual simple recurrent network (DSRN) model that includes a surface SRN encoding and predicting the surface properties of stimuli and an abstract SRN encoding and predicting the abstract properties of stimuli. The results of Simulations 1 and 2 showed that the DSRN model can account for learning effects in the serial reaction time (SRT) task under different conditions, and the manipulation of the contribution weight of each SRN accounted for the contribution of conscious and unconscious processes in inclusion and exclusion tests in previous studies. The results of human performance in Simulation 3 provided further evidence that people can implicitly learn both chunking and abstract knowledge in sequence learning, and the results of Simulation 3 confirmed that the DSRN model can account for how people implicitly acquire the two types of knowledge in sequence learning. These findings extend the learning ability of the SRN model and help understand how different types of knowledge can be acquired implicitly in sequence learning.

Simulations of implicit learning of symmetries: The importance of prior knowledge and the nature of the memory buffer

The current research aimed to investigate the role that prior knowledge played in what structures could be implicitly learnt and also the nature of the memory buffer required for learning such structures. It is already established that people can implicitly learn to detect an inversion symmetry (i.e. a cross-serial dependency) based on linguistic tone types. The present study investigated the ability of the Simple Recurrent Network (SRN) to explain implicit learning of such recursive structures. We found that the SRN learnt the symmetry over tone types more effectively when given prior knowledge of the tone types (i.e. of the two categories tones were grouped into). The role of prior knowledge of the tone types in learning the inversion symmetry was tested on people: When an arbitrary classification of tones was used (i.e. in the absence of prior knowledge of categories), participants did not implicitly learn the inversion symmetry (unlike when they did have prior knowledge of the tone types). These results indicate the importance of prior knowledge in implicit learning of symmetrical structures. We further contrasted the learning of inversion symmetry and retrograde symmetry and showed that inversion was learnt more easily than retrograde by the SRN, matching our previous findings with people, thus showing that the type of memory buffer used in the SRN is suitable for modeling the implicit learning of symmetry in people.

Cognitive Science Honors the Memory of Jeffrey Elman

Jeff Elman (1/22/1948–6/28/2018) was a major and much beloved figure in cognitive science, best known for his work on the TRACE model of speech perception, simple recurrent network models of the temporal dynamics of language processing, and his coauthored monograph, Rethinking Innateness. Beyond his individual and collaborative research, he is widely recognized for his lasting contributions to building our scientific community. Here we celebrate his contributions by briefly recounting his life’s work and sharing commentaries and reminiscences from a number of his closest colleagues over the years.

Pre-Wiring and Pre-Training: What Does a Neural Network Need to Learn Truly General Identity Rules?

In an influential paper (“Rule Learning by Seven-Month-Old Infants”), Marcus, Vijayan, Rao and Vishton claimed that connectionist models cannot account for human success at learning tasks that involved generalization of abstract knowledge such as grammatical rules. This claim triggered a heated debate, centered mostly around variants of the Simple Recurrent Network model. In our work, we revisit this unresolved debate and analyze the underlying issues from a different perspective. We argue that, in order to simulate human-like learning of grammatical rules, a neural network model should not be used as a tabula rasa, but rather, the initial wiring of the neural connections and the experience acquired prior to the actual task should be incorporated into the model. We present two methods that aim to provide such initial state: a manipulation of the initial connections of the network in a cognitively plausible manner (concretely, by implementing a “delay-line” memory), and a pre-training algorithm that incrementally challenges the network with novel stimuli. We implement such techniques in an Echo State Network (ESN), and we show that only when combining both techniques the ESN is able to learn truly general identity rules. Finally, we discuss the relation between these cognitively motivated techniques and recent advances in Deep Learning.

Learning Simpler Language Models with the Differential State Framework

Learning useful information across long time lags is a critical and difficult problem for temporal neural models in tasks such as language modeling. Existing architectures that address the issue are often complex and costly to train. The differential state framework (DSF) is a simple and high-performing design that unifies previously introduced gated neural models. DSF models maintain longer-term memory by learning to interpolate between a fast-changing data-driven representation and a slowly changing, implicitly stable state. Within the DSF framework, a new architecture is presented, the delta-RNN. This model requires hardly any more parameters than a classical, simple recurrent network. In language modeling at the word and character levels, the delta-RNN outperforms popular complex architectures, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU), and, when regularized, performs comparably to several state-of-the-art baselines. At the subword level, the delta-RNN's performance is comparable to that of complex gated architectures.