scholarly journals Entorhinal and ventromedial prefrontal cortices abstract and generalize the structure of reinforcement learning problems

Neuron ◽  
2020 ◽  
Author(s):  
Alon Boaz Baram ◽  
Timothy Howard Muller ◽  
Hamed Nili ◽  
Mona Maria Garvert ◽  
Timothy Edward John Behrens
Keyword(s):  
Reinforcement Learning ◽  
Learning Problems ◽  
Prefrontal Cortices

scholarly journals Entorhinal and ventromedial prefrontal cortices abstract and generalise the structure of reinforcement learning problems

2019 ◽  
Author(s):  
Alon B Baram ◽  
Timothy H Muller ◽  
Hamed Nili ◽  
Mona Garvert ◽  
Timothy E J Behrens
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Spatial Reasoning ◽  
Ventromedial Prefrontal Cortex ◽  
Learning Problems ◽  
Structural Knowledge ◽  
Task Structure ◽  
Rapid Learning ◽  
Flexible Inference ◽  
Prefrontal Cortices

AbstractKnowledge of the structure of a problem, such as relationships between stimuli, enables rapid learning and flexible inference. Humans and other animals can abstract this structural knowledge and generalise it to solve new problems. For example, in spatial reasoning, shortest-path inferences are immediate in new environments. Spatial structural transfer is mediated by grid cells in entorhinal and (in humans) medial prefrontal cortices, which maintain their structure across different environments. Here, using fMRI, we show that entorhinal and ventromedial prefrontal cortex (vmPFC) representations perform a much broader role in generalising the structure of problems. We introduce a task-remapping paradigm, where subjects solve multiple reinforcement learning (RL) problems differing in structural or sensory properties. We show that, as with space, entorhinal representations are preserved across different RL problems only if task structure is preserved. In vmPFC, representations of standard RL signals such as prediction error also vary as a function of task structure.

Learning and control

2018 ◽  
Author(s):  
Ivan Herreros
Keyword(s):  
Machine Learning ◽  
Reinforcement Learning ◽  
Brain Function ◽  
Control Strategies ◽  
Learning Problems ◽  
Animal Learning ◽  
Feed Forward Control ◽  
Machine Learning Applications ◽  
And Control

This chapter discusses basic concepts from control theory and machine learning to facilitate a formal understanding of animal learning and motor control. It first distinguishes between feedback and feed-forward control strategies, and later introduces the classification of machine learning applications into supervised, unsupervised, and reinforcement learning problems. Next, it links these concepts with their counterparts in the domain of the psychology of animal learning, highlighting the analogies between supervised learning and classical conditioning, reinforcement learning and operant conditioning, and between unsupervised and perceptual learning. Additionally, it interprets innate and acquired actions from the standpoint of feedback vs anticipatory and adaptive control. Finally, it argues how this framework of translating knowledge between formal and biological disciplines can serve us to not only structure and advance our understanding of brain function but also enrich engineering solutions at the level of robot learning and control with insights coming from biology.

Hierarchical Reinforcement Learning

2011 ◽  
pp. 825-830
Author(s):  
Carlos Diuk ◽  
Michael Littman
Keyword(s):  
Reinforcement Learning ◽  
Learning Problems ◽  
Underlying Structure ◽  
Sequential Decision ◽  
State Spaces ◽  
Hierarchical Reinforcement Learning ◽  
Markov Decision ◽  
Finite Set ◽  
State Abstraction ◽  
Main Ideas

Reinforcement learning (RL) deals with the problem of an agent that has to learn how to behave to maximize its utility by its interactions with an environment (Sutton & Barto, 1998; Kaelbling, Littman & Moore, 1996). Reinforcement learning problems are usually formalized as Markov Decision Processes (MDP), which consist of a finite set of states and a finite number of possible actions that the agent can perform. At any given point in time, the agent is in a certain state and picks an action. It can then observe the new state this action leads to, and receives a reward signal. The goal of the agent is to maximize its long-term reward. In this standard formalization, no particular structure or relationship between states is assumed. However, learning in environments with extremely large state spaces is infeasible without some form of generalization. Exploiting the underlying structure of a problem can effect generalization and has long been recognized as an important aspect in representing sequential decision tasks (Boutilier et al., 1999). Hierarchical Reinforcement Learning is the subfield of RL that deals with the discovery and/or exploitation of this underlying structure. Two main ideas come into play in hierarchical RL. The first one is to break a task into a hierarchy of smaller subtasks, each of which can be learned faster and easier than the whole problem. Subtasks can also be performed multiple times in the course of achieving the larger task, reusing accumulated knowledge and skills. The second idea is to use state abstraction within subtasks: not every task needs to be concerned with every aspect of the state space, so some states can actually be abstracted away and treated as the same for the purpose of the given subtask.

ANALYTICAL REVIEW OF MULTI-AGENT REINFORCEMENT LEARNING PROBLEMS

2020 ◽  
pp. 48-56
Author(s):  
Yu. V. Dubenko
Keyword(s):  
Reinforcement Learning ◽  
Intelligent Agents ◽  
Russian Language ◽  
Learning Problems ◽  
Multi Agent Systems ◽  
Hierarchical Reinforcement Learning ◽  
Collective Interaction ◽  
Analytical Review ◽  
Multi Agent ◽  
Partially Observable

This paper is devoted to the problem of collective artificial intelligence in solving problems by intelligent agents in external environments. The environments may be: fully or partially observable, deterministic or stochastic, static or dynamic, discrete or continuous. The paper identifies problems of collective interaction of intelligent agents when they solve a class of tasks, which need to coordinate actions of agent group, e. g. task of exploring the territory of a complex infrastructure facility. It is revealed that the problem of reinforcement training in multi-agent systems is poorly presented in the press, especially in Russian-language publications. The article analyzes reinforcement learning, describes hierarchical reinforcement learning, presents basic methods to implement reinforcement learning. The concept of macro-action by agents integrated in groups is introduced. The main problems of intelligent agents collective interaction for problem solving (i. e. calculation of individual rewards for each agent; agent coordination issues; application of macro actions by agents integrated into groups; exchange of experience generated by various agents as part of solving a collective problem) are identified. The model of multi-agent reinforcement learning is described in details. The article describes problems of this approach building on existing solutions. Basic problems of multi-agent reinforcement learning are formulated in conclusion.

Sign in / Sign up

Export Citation Format

Share Document