scholarly journals Compositional clustering in task structure learning

2017 ◽  
Author(s):  
Nicholas Franklin ◽  
Michael J. Frank
Keyword(s):  
Computational Models ◽  
Structure Learning ◽  
Transition Function ◽  
Task Structure ◽  
Reward Function ◽  
Transition Functions ◽  
Learning Agent ◽  
Many Instruments ◽  
Reward Functions ◽  
New Policies

AbstractHumans are remarkably adept at generalizing knowledge between experiences in a way that can be difficult for computers. Often, this entails generalizing constituent pieces of experiences that do not fully overlap, but nonetheless share useful similarities with, previously acquired knowledge. However, it is often unclear how knowledge gained in one context should generalize to another. Previous computational models and data suggest that rather than learning about each individual context, humans build latent abstract structures and learn to link these structures to arbitrary contexts, facilitating generalization. In these models, task structures that are more popular across contexts are more likely to be revisited in new contexts. However, these models can only re-use policies as a whole and are unable to transfer knowledge about the transition structure of the environment even if only the goal has changed (or vice-versa). This contrasts with ecological settings, where some aspects of task structure, such as the transition function, will be shared between context separately from other aspects, such as the reward function. Here, we develop a novel non-parametric Bayesian agent that forms independent latent clusters for transition and reward functions, affording separable transfer of their constituent parts across contexts. We show that the relative performance of this agent compared to an agent that jointly clusters reward and transition functions depends environmental task statistics: the mutual information between transition and reward functions and the stochasticity of the observations. We formalize our analysis through an information theoretic account of the priors, and propose a meta learning agent that dynamically arbitrates between strategies across task domains to optimize a statistical tradeoff.Author summaryA musician may learn to generalize behaviors across instruments for different purposes, for example, reusing hand motions used when playing classical on the flute to play jazz on the saxophone. Conversely, she may learn to play a single song across many instruments that require completely distinct physical motions, but nonetheless transfer knowledge between them. This degree of compositionality is often absent from computational frameworks of learning, forcing agents either to generalize entire learned policies or to learn new policies from scratch. Here, we propose a solution to this problem that allows an agent to generalize components of a policy independently and compare it to an agent that generalizes components as a whole. We show that the degree to which one form of generalization is favored over the other is dependent on the features of task domain, with independent generalization of task components favored in environments with weak relationships between components or high degrees of noise and joint generalization of task components favored when there is a clear, discoverable relationship between task components. Furthermore, we show that the overall meta structure of the environment can be learned and leveraged by an agent that dynamically arbitrates between these forms of structure learning.

scholarly journals Reward-predictive representations generalize across tasks in reinforcement learning

2019 ◽  
Author(s):  
Lucas Lehnert ◽  
Michael L. Littman ◽  
Michael J. Frank
Keyword(s):  
Reinforcement Learning ◽  
Transfer Characteristic ◽  
Transition Function ◽  
Equivalence Relations ◽  
State Representation ◽  
Transition Functions ◽  
Right Hand ◽  
Learning Agent ◽  
Biological Studies ◽  
Reward Functions

AbstractIn computer science, reinforcement learning is a powerful framework with which artificial agents can learn to maximize their performance for any given Markov decision process (MDP). Advances over the last decade, in combination with deep neural networks, have enjoyed performance advantages over humans in many difficult task settings. However, such frameworks perform far less favorably when evaluated in their ability to generalize or transfer representations across different tasks. Existing algorithms that facilitate transfer typically are limited to cases in which the transition function or the optimal policy is portable to new contexts, but achieving “deep transfer” characteristic of human behavior has been elusive. Such transfer typically requires discovery of abstractions that permit analogical reuse of previously learned representations to superficially distinct tasks. Here, we demonstrate that abstractions that minimize error in predictions of reward outcomes generalize across tasks with different transition and reward functions. Such reward-predictive representations compress the state space of a task into a lower dimensional representation by combining states that are equivalent in terms of both the transition and reward functions. Because only state equivalences are considered, the resulting state representation is not tied to the transition and reward functions themselves and thus generalizes across tasks with different reward and transition functions. These results contrast with those using abstractions that myopically maximize reward in any given MDP and motivate further experiments in humans and animals to investigate if neural and cognitive systems involved in state representation perform abstractions that facilitate such equivalence relations.Author summaryHumans are capable of transferring abstract knowledge from one task to another. For example, in a right-hand-drive country, a driver has to use the right arm to operate the shifter. A driver who learned how to drive in a right-hand-drive country can adapt to operating a left-hand-drive car and use the other arm for shifting instead of re-learning how to drive. Despite the fact that both tasks require different coordination of motor skills, both tasks are the same in an abstract sense: In both tasks, a car is operated and there is the same progression from 1st to 2nd gear and so on. We study distinct algorithms by which a reinforcement learning agent can discover state representations that encode knowledge about a particular task, and evaluate how well they can generalize. Through a sequence of simulation results, we show that state abstractions that minimize errors in prediction about future reward outcomes generalize across tasks, even those that superficially differ in both the goals (rewards) and the transitions from one state to the next. This work motivates biological studies to determine if distinct circuits are adapted to maximize reward vs. to discover useful state representations.

scholarly journals What if the World Were Different? Gradient-Based Exploration for New Optimal Policies

10.29007/6jsv ◽  
2018 ◽  
Author(s):  
Rui Silva ◽  
Francisco S. Melo ◽  
Manuela Veloso
Keyword(s):  
Optimal Policy ◽  
Optimization Problem ◽  
Possible Worlds ◽  
Practical Interest ◽  
Transition Function ◽  
Planning Under Uncertainty ◽  
Reward Function ◽  
The World ◽  
Gradient Based ◽  
New Policies

Planning under uncertainty assumes a model of the world that specifies the probabilistic effects of the actions of an agent in terms of changes of the state. Given such model, planning proceeds to determine a policy that defines for each state the choice of action that the agent should follow in order to maximize a reward function. In this work, we realize that the world can be changed in more ways than those possible by the execution of the agent’s repertoire of actions. These additional configurations of the world may allow new policies that let the agent accumulate even more reward than that possible by following the optimal policy of the original world. We introduce and formalize the problem of planning while considering these additional possible worlds. We then present an approach that models feasible changes to the world as modifications to the probability transition function, and show that the problem of computing the configuration of the world that allows the most rewarding optimal policy can be formulated as a constrained optimization problem. Finally, we contribute a gradient-based algorithm for solving this optimization problem. Experimental evaluation shows the effectiveness of our approach in multiple problems of practical interest.

scholarly journals Generalizing to generalize: when (and when not) to be compositional in task structure learning

2019 ◽  
Author(s):  
Nicholas T. Franklin ◽  
Michael J. Frank
Keyword(s):  
National Science Foundation ◽  
Structure Learning ◽  
Human Subjects ◽  
Empirical Studies ◽  
Theoretical Work ◽  
State Transitions ◽  
Task Structure ◽  
Learned Behavior ◽  
Learning Agent ◽  
Author Note

AbstractHumans routinely face novel environments in which they have to generalize in order toact adaptively. However, doing so involves the non-trivial challenge of deciding which aspects of a task domain to generalize. While it is sometimes appropriate to simply re-use a learned behavior, often adaptive generalization entails recombining distinct components of knowledge acquired across multiple contexts. Theoretical work has suggested a computational trade-off in which it can be more or less useful to learn and generalize aspects of task structure jointly or compositionally, depending on previous task statistics, but empirical studies are lacking. Here we develop a series of navigation tasks which manipulate the statistics of goal values (“what to do”) and state transitions (“how to do it”) across contexts, and assess whether human subjects generalize these task components separately or conjunctively. We find that human generalization is sensitive to the statistics of the previously experienced task domain, favoring compositional or conjunctive generalization when the task statistics are indicative of such structures, and a mixture of the two when they are more ambiguous. These results support the predictions of a normative “meta-generalization learning” agent that does not only generalize previous knowledge but also generalizes the statistical structure most likely to support generalization.Author NoteThis work was supported in part by the National Science Foundation Proposal 1460604 “How Prefrontal Cortex Augments Reinforcement Learning” to MJF. We thank Mark Ho for providing code used in the behavioral task. We thank Matt Nassar for helpful discussions. Correspondence should be addressed to Nicholas T. Franklin ([email protected]) or Michael J. Frank ([email protected]).

scholarly journals Efficient PAC Reinforcement Learning in Regular Decision Processes

2021 ◽  
Author(s):  
Alessandro Ronca ◽  
Giuseppe De Giacomo
Keyword(s):  
Reinforcement Learning ◽  
Markov Decision Process ◽  
Polynomial Time ◽  
Optimal Policy ◽  
Decision Process ◽  
Transition Function ◽  
Decision Processes ◽  
Reward Function ◽  
Markov Decision ◽  
Reward Functions

Recently regular decision processes have been proposed as a well-behaved form of non-Markov decision process. Regular decision processes are characterised by a transition function and a reward function that depend on the whole history, though regularly (as in regular languages). In practice both the transition and the reward functions can be seen as finite transducers. We study reinforcement learning in regular decision processes. Our main contribution is to show that a near-optimal policy can be PAC-learned in polynomial time in a set of parameters that describe the underlying decision process. We argue that the identified set of parameters is minimal and it reasonably captures the difficulty of a regular decision process.

scholarly journals CURIOSITY-DRIVEN REINFORCEMENT LEARNING AGENT FOR MAPPING UNKNOWN INDOOR ENVIRONMENTS

2021 ◽  
Vol V-1-2021 ◽  
pp. 129-136
Author(s):  
N. Botteghi ◽  
R. Schulte ◽  
B. Sirmacek ◽  
M. Poel ◽  
C. Brune
Keyword(s):  
Artificial Intelligence ◽  
Reinforcement Learning ◽  
Mobile Robot ◽  
Indoor Environments ◽  
Reward Function ◽  
Learning Framework ◽  
Learning Agent ◽  
The World ◽  
Reward Functions ◽  
Slam Algorithm

Abstract. Autonomously exploring and mapping is one of the open challenges of robotics and artificial intelligence. Especially when the environments are unknown, choosing the optimal navigation directive is not straightforward. In this paper, we propose a reinforcement learning framework for navigating, exploring, and mapping unknown environments. The reinforcement learning agent is in charge of selecting the commands for steering the mobile robot, while a SLAM algorithm estimates the robot pose and maps the environments. The agent, to select optimal actions, is trained to be curious about the world. This concept translates into the introduction of a curiosity-driven reward function that encourages the agent to steer the mobile robot towards unknown and unseen areas of the world and the map. We test our approach in explorations challenges in different indoor environments. The agent trained with the proposed reward function outperforms the agents trained with reward functions commonly used in the literature for solving such tasks.

scholarly journals Cooperative update of beliefs and state-transition functions in human reinforcement learning

2019 ◽  
Vol 9 (1) ◽  
Author(s):  
Hiroshi Higashi ◽  
Tetsuto Minami ◽  
Shigeki Nakauchi
Keyword(s):  
Reinforcement Learning ◽  
Bayesian Learning ◽  
Computational Models ◽  
State Transition ◽  
Learning Strategy ◽  
The State ◽  
Reward Function ◽  
Outcome Feedback ◽  
Transition Functions ◽  
Trial Analysis

AbstractIt is widely known that reinforcement learning systems in the brain contribute to learning via interactions with the environment. These systems are capable of solving multidimensional problems, in which some dimensions are relevant to a reward, while others are not. To solve these problems, computational models use Bayesian learning, a strategy supported by behavioral and neural evidence in human. Bayesian learning takes into account beliefs, which represent a learner’s confidence in a particular dimension being relevant to the reward. Beliefs are given as a posterior probability of the state-transition (reward) function that maps the optimal actions to the states in each dimension. However, when it comes to implementing this learning strategy, the order in which beliefs and state-transition functions update remains unclear. The present study investigates this update order using a trial-by-trial analysis of human behavior and electroencephalography signals during a task in which learners have to identify the reward-relevant dimension. Our behavioral and neural results reveal a cooperative update—within 300 ms after the outcome feedback, the state-transition functions are updated, followed by the beliefs for each dimension.

scholarly journals Spontaneous task structure formation results in a cost to incidental memory of task stimuli

2018 ◽  
Author(s):  
Christina Bejjani ◽  
Tobias Egner
Keyword(s):  
Structure Formation ◽  
Structure Learning ◽  
Learning Task ◽  
Underlying Structure ◽  
Task Structure ◽  
Cognitive Demands ◽  
Task Set ◽  
Stimulus Response ◽  
Attention To Task ◽  
Task Sets

Humans are characterized by their ability to leverage rules for classifying and linking stimuli to context-appropriate actions. Previous studies have shown that when humans learn stimulus-response associations for two-dimensional stimuli, they implicitly form and generalize hierarchical rule structures (task-sets). However, the cognitive processes underlying structure formation are poorly understood. Across four experiments, we manipulated how trial-unique images mapped onto responses to bias spontaneous task-set formation and investigated structure learning through the lens of incidental stimulus encoding. Participants performed a learning task designed to either promote task-set formation (by “motor-clustering” possible stimulus-action rules), or to discourage it (by using arbitrary category-response mappings). We adjudicated between two hypotheses: Structure learning may promote attention to task stimuli, thus resulting in better subsequent memory. Alternatively, building task-sets might impose cognitive demands (for instance, on working memory) that divert attention away from stimulus encoding. While the clustering manipulation affected task-set formation, there were also substantial individual differences. Importantly, structure learning incurred a cost: spontaneous task-set formation was associated with diminished stimulus encoding. Thus, spontaneous hierarchical task-set formation appears to involve cognitive demands that divert attention away from encoding of task stimuli during structure learning.

Evaluating the learning and performance characteristics of self-organizing systems with different task features

2021 ◽  
pp. 1-19
Author(s):  
Hao Ji ◽  
Yan Jin
Keyword(s):  
Systematic Evaluation ◽  
Full Potential ◽  
Learning Capability ◽  
Reward Function ◽  
Reward Functions ◽  
Good Learning ◽  
And Performance ◽  
Generation Problem ◽  
Self Organizing

Abstract Self-organizing systems (SOS) are developed to perform complex tasks in unforeseen situations with adaptability. Predefining rules for self-organizing agents can be challenging, especially in tasks with high complexity and changing environments. Our previous work has introduced a multiagent reinforcement learning (RL) model as a design approach to solving the rule generation problem of SOS. A deep multiagent RL algorithm was devised to train agents to acquire the task and self-organizing knowledge. However, the simulation was based on one specific task environment. Sensitivity of SOS to reward functions and systematic evaluation of SOS designed with multiagent RL remain an issue. In this paper, we introduced a rotation reward function to regulate agent behaviors during training and tested different weights of such reward on SOS performance in two case studies: box-pushing and T-shape assembly. Additionally, we proposed three metrics to evaluate the SOS: learning stability, quality of learned knowledge, and scalability. Results show that depending on the type of tasks; designers may choose appropriate weights of rotation reward to obtain the full potential of agents’ learning capability. Good learning stability and quality of knowledge can be achieved with an optimal range of team sizes. Scaling up to larger team sizes has better performance than scaling downwards.

scholarly journals Optimal Stopping for Processes with Independent Increments, and Applications

2009 ◽  
Vol 46 (04) ◽  
pp. 1130-1145 ◽  
Author(s):  
G. Deligiannidis ◽  
H. Le ◽  
S. Utev
Keyword(s):  
Optimal Stopping ◽  
Optimization Problems ◽  
Infinite Horizon ◽  
Unified Approach ◽  
Reward Function ◽  
Finance Industry ◽  
Independent Increments ◽  
Reward Functions ◽  
First Time ◽  
The Value Function

In this paper we present an explicit solution to the infinite-horizon optimal stopping problem for processes with stationary independent increments, where reward functions admit a certain representation in terms of the process at a random time. It is shown that it is optimal to stop at the first time the process crosses a level defined as the root of an equation obtained from the representation of the reward function. We obtain an explicit formula for the value function in terms of the infimum and supremum of the process, by making use of the Wiener–Hopf factorization. The main results are applied to several problems considered in the literature, to give a unified approach, and to new optimization problems from the finance industry.

scholarly journals Multi-Zone Unit for Recurrent Neural Networks

2020 ◽  
Vol 34 (04) ◽  
pp. 5150-5157
Author(s):  
Fandong Meng ◽  
Jinchao Zhang ◽  
Yang Liu ◽  
Jie Zhou
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Critical Role ◽  
Transition Function ◽  
Learning Problems ◽  
Arbitrary Length ◽  
Transition Functions ◽  
Multiple Datasets ◽  
Analysis Task ◽  
Hidden States

Recurrent neural networks (RNNs) have been widely used to deal with sequence learning problems. The input-dependent transition function, which folds new observations into hidden states to sequentially construct fixed-length representations of arbitrary-length sequences, plays a critical role in RNNs. Based on single space composition, transition functions in existing RNNs often have difficulty in capturing complicated long-range dependencies. In this paper, we introduce a new Multi-zone Unit (MZU) for RNNs. The key idea is to design a transition function that is capable of modeling multiple space composition. The MZU consists of three components: zone generation, zone composition, and zone aggregation. Experimental results on multiple datasets of the character-level language modeling task and the aspect-based sentiment analysis task demonstrate the superiority of the MZU.

Sign in / Sign up

Export Citation Format

Share Document