Can Recurrent Neural Networks Learn Nested Recursion?

Context-free grammars (CFG) were one of the first formal tools used to model natural languages, and they remain relevant today as the basis of several frameworks. A key ingredient of CFG is the presence of nested recursion. In this paper, we investigate experimentally the capability of several recurrent neural networks (RNNs) to learn nested recursion. More precisely, we measure an upper bound of their capability to do so, by simplifying the task to learning a generalized Dyck language, namely one composed of matching parentheses of various kinds. To do so, we present the RNNs with a set of random strings having a given maximum nesting depth and test its ability to predict the kind of closing parenthesis when facing deeper nested strings. We report mixed results: when generalizing to deeper nesting levels, the accuracy of standard RNNs is significantly higher than random, but still far from perfect. Additionally, we propose some non-standard stack-based models which can approach perfect accuracy, at the cost of robustness.

Nested Recursions, Simultaneous Parameters and Tree Superpositions

We apply a tree-based methodology to solve new, very broadly defined families of nested recursions of the general form $R(n)=\sum_{t=1}^k R(n-a_t-\sum_{i=1}^{p}R(n-b_{ti}))$, where $a_t$ are integers, $b_{ti}$ are natural numbers, and $k,p$ are natural numbers that we use to denote "arity" and "order," respectively, and with some specified initial conditions. The key idea of the tree-based solution method is to associate such recursions with infinite labelled trees in a natural way so that the solution to the recursions solves a counting question relating to the corresponding trees. We characterize certain recursion families within $R(n)$ by introducing "simultaneous parameters" that appear both within the recursion itself and that also specify structural properties of the corresponding tree. First, we extend and unify recently discovered results concerning two families of arity $k=2$, order $p=1$ recursions. Next, we investigate the solution of nested recursion families by taking linear combinations of solution sequence frequencies for simpler nested recursions, which correspond to superpositions of the associated trees; this leads us to identify and solve two new recursion families for arity $k=2$ and general order $p$. Finally, we extend these results to general arity $k>2$. We conclude with several related open problems.