scholarly journals A common deliberative process underlies model-based planning and patient intertemporal choice

2018 ◽  
Author(s):  
Lindasy E Hunter ◽  
Aaron M Bornstein ◽  
Catherine A Hartley
Keyword(s):  
Reinforcement Learning ◽  
Intertemporal Choice ◽  
Learning Task ◽  
Self Control ◽  
Common Mechanism ◽  
Deliberative Process ◽  
Model Based ◽  
Near Term ◽  
Economic Measure ◽  
Core Construct

Humans and animals consistently forego, or 'discount' future rewards in favor of more proximal, but less valuable, options. This behavior is often thought of in terms of a failure of 'self-control', a lack of inhibition when considering the possibility of immediate gratification. However, rather than overweighting the near-term reward, the same behavior can result from failing to properly consider the far-off reward. The capacity to plan for future gains is a core construct in Reinforcement Learning (RL), known as 'model-based' planning. Both discounting and model-based planning have been shown to track everyday behaviors from diet to exercise habits to drug abuse. Here, we show that these two capacities are related via a common mechanism -- people who are more likely to deliberate about future reward in an intertemporal choice task, as indicated by the time they spend considering the choice, are also more likely to make multi-step plans for reward in a sequential reinforcement learning task. In contrast, the degree to which people's intertemporal choices were driven by a more automatic bias did not correspond to their planning tendency, and neither did the more standard measure of discounting behavior. These results suggest that the standard behavioral economic measure of discounting is more fruitfully understood by decomposing it into constituent parts, and that only one of these parts corresponds to the sort of multi-step thinking needed to make plans for the future.

Model-Based and Model-Free Social Cognition

2019 ◽  
Author(s):  
Leor M Hackel ◽  
Jeffrey Jordan Berg ◽  
Björn Lindström ◽  
David Amodio
Keyword(s):  
Reinforcement Learning ◽  
Social Cognition ◽  
Learning Strategies ◽  
Memory Systems ◽  
Learning Task ◽  
Financial Advisors ◽  
Model Based ◽  
Model Free ◽  
Systems Model ◽  
Task Assessment

Do habits play a role in our social impressions? To investigate the contribution of habits to the formation of social attitudes, we examined the roles of model-free and model-based reinforcement learning in social interactions—computations linked in past work to habit and planning, respectively. Participants in this study learned about novel individuals in a sequential reinforcement learning paradigm, choosing financial advisors who led them to high- or low-paying stocks. Results indicated that participants relied on both model-based and model-free learning, such that each independently predicted choice during the learning task and self-reported liking in a post-task assessment. Specifically, participants liked advisors who could provide large future rewards as well as advisors who had provided them with large rewards in the past. Moreover, participants varied in their use of model-based and model-free learning strategies, and this individual difference influenced the way in which learning related to self-reported attitudes: among participants who relied more on model-free learning, model-free social learning related more to post-task attitudes. We discuss implications for attitudes, trait impressions, and social behavior, as well as the role of habits in a memory systems model of social cognition.

scholarly journals Predictive representations can link model-based reinforcement learning to model-free mechanisms

2016 ◽  
Author(s):  
Evan M. Russek ◽  
Ida Momennejad ◽  
Matthew M. Botvinick ◽  
Samuel J. Gershman ◽  
Nathaniel D. Daw
Keyword(s):  
Reinforcement Learning ◽  
Neural Circuits ◽  
Full Range ◽  
Decision Time ◽  
Empirical Literature ◽  
Automatic Process ◽  
Deliberative Process ◽  
Model Based ◽  
Model Free ◽  
Long Run

AbstractHumans and animals are capable of evaluating actions by considering their long-run future rewards through a process described using model-based reinforcement learning (RL) algorithms. The mechanisms by which neural circuits perform the computations prescribed by model-based RL remain largely unknown; however, multiple lines of evidence suggest that neural circuits supporting model-based behavior are structurally homologous to and overlapping with those thought to carry out model-free temporal difference (TD) learning. Here, we lay out a family of approaches by which model-based computation may be built upon a core of TD learning. The foundation of this framework is the successor representation, a predictive state representation that, when combined with TD learning of value predictions, can produce a subset of the behaviors associated with model-based learning, while requiring less decision-time computation than dynamic programming. Using simulations, we delineate the precise behavioral capabilities enabled by evaluating actions using this approach, and compare them to those demonstrated by biological organisms. We then introduce two new algorithms that build upon the successor representation while progressively mitigating its limitations. Because this framework can account for the full range of observed putatively model-based behaviors while still utilizing a core TD framework, we suggest that it represents a neurally plausible family of mechanisms for model-based evaluation.Author SummaryAccording to standard models, when confronted with a choice, animals and humans rely on two separate, distinct processes to come to a decision. One process deliberatively evaluates the consequences of each candidate action and is thought to underlie the ability to flexibly come up with novel plans. The other process gradually increases the propensity to perform behaviors that were previously successful and is thought to underlie automatically executed, habitual reflexes. Although computational principles and animal behavior support this dichotomy, at the neural level, there is little evidence supporting a clean segregation. For instance, although dopamine — famously implicated in drug addiction and Parkinson’s disease — currently only has a well-defined role in the automatic process, evidence suggests that it also plays a role in the deliberative process. In this work, we present a computational framework for resolving this mismatch. We show that the types of behaviors associated with either process could result from a common learning mechanism applied to different strategies for how populations of neurons could represent candidate actions. In addition to demonstrating that this account can produce the full range of flexible behavior observed in the empirical literature, we suggest experiments that could detect the various approaches within this framework.

Neuroevolution approach for reinforcement learning task applied to control system of a cart pole system

2019 ◽  
Author(s):  
Marco Boaretto ◽  
Gabriel Chaves Becchi ◽  
Luiza Scapinello Aquino ◽  
Aderson Cleber Pifer ◽  
Helon Vicente Hultmann Ayala ◽  
...  
Keyword(s):  
Control System ◽  
Reinforcement Learning ◽  
Learning Task ◽  
Pole System

Bestärkendes Lernen mittels Offline-Trajektorienplanung basierend auf iterativ approximierten Modellen

2020 ◽  
Vol 68 (8) ◽  
pp. 612-624
Author(s):  
Max Pritzkoleit ◽  
Robert Heedt ◽  
Carsten Knoll ◽  
Klaus Röbenack
Keyword(s):  
Reinforcement Learning ◽  
Neuronale Netze ◽  
Model Based ◽  
Künstliche Neuronale Netze

ZusammenfassungIn diesem Beitrag nutzen wir Künstliche Neuronale Netze (KNN) zur Approximation der Dynamik nichtlinearer (mechanischer) Systeme. Diese iterativ approximierten neuronalen Systemmodelle werden in einer Offline-Trajektorienplanung verwendet, um eine optimale Rückführung zu bestimmen, welche auf das reale System angewandt wird. Dieser Ansatz des modellbasierten bestärkenden Lernens (engl. model-based reinforcement learning (RL)) wird am Aufschwingen des Einfachwagenpendels zunächst simulativ evaluiert und zeigt gegenüber modellfreien RL-Ansätzen eine signifikante Verbesserung der Dateneffizienz. Weiterhin zeigen wir Experimentalergebnisse an einem Versuchsstand, wobei der vorgestellte Algorithmus innerhalb weniger Versuche in der Lage ist, eine für das System optimale Rückführung hinreichend gut zu approximieren.

scholarly journals Uncertainty-aware Contact-safe Model-based Reinforcement Learning

2021 ◽  
pp. 1-1
Author(s):  
Cheng-Yu Kuo ◽  
Andreas Schaarschmidt ◽  
Yunduan Cui ◽  
Tamim Asfour ◽  
Takamitsu Matsubara
Keyword(s):  
Reinforcement Learning ◽  
Model Based ◽  
Safe Model
Sign in / Sign up

Export Citation Format

Share Document