scholarly journals L-DOPA Reduces Model-Free Control of Behavior by Attenuating the Transfer of Value to Action

2016 ◽  
Author(s):  
Nils B. Kroemer ◽  
Ying Lee ◽  
Shakoor Pooseh ◽  
Ben Eppinger ◽  
Thomas Goschke ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Prediction Error ◽  
Behavioral Control ◽  
Decision Task ◽  
Learning Models ◽  
Model Free ◽  
Markov Decision ◽  
Model Free Control ◽  
Reinforcement Learning Models ◽  
The Brain

AbstractDopamine is a key neurotransmitter in reinforcement learning and action control. Recent findings suggest that these components are inherently entangled. Here, we tested if increases in dopamine tone by administration of L-DOPA upregulate deliberative “model-based” control of behavior or reflexive “model-free” control as predicted by dual-control reinforcement-learning models. Alternatively, L-DOPA may impair learning as suggested by “value” or “thrift” theories of dopamine. To this end, we employed a two-stage Markov decision-task to investigate the effect of L-DOPA (randomized cross-over) on behavioral control while brain activation was measured using fMRI. L-DOPA led to attenuated model-free control of behavior as indicated by the reduced impact of reward on choice and increased stochasticity of model-free choices. Correspondingly, in the brain, L-DOPA decreased the effect of reward while prediction-error signals were unaffected. Taken together, our results suggest that L-DOPA reduces model-free control of behavior by attenuating the transfer of value to action.

scholarly journals Prefrontal solution to the bias-variance tradeoff during reinforcement learning

2020 ◽  
Author(s):  
Dongjae Kim ◽  
Jaeseung Jeong ◽  
Sang Wan Lee
Keyword(s):  
Adaptive Control ◽  
Reinforcement Learning ◽  
Prediction Error ◽  
Brain Regions ◽  
Decision Task ◽  
Prediction Errors ◽  
Model Based ◽  
Model Free ◽  
Bias Variance ◽  
The Brain

AbstractThe goal of learning is to maximize future rewards by minimizing prediction errors. Evidence have shown that the brain achieves this by combining model-based and model-free learning. However, the prediction error minimization is challenged by a bias-variance tradeoff, which imposes constraints on each strategy’s performance. We provide new theoretical insight into how this tradeoff can be resolved through the adaptive control of model-based and model-free learning. The theory predicts the baseline correction for prediction error reduces the lower bound of the bias–variance error by factoring out irreducible noise. Using a Markov decision task with context changes, we showed behavioral evidence of adaptive control. Model-based behavioral analyses show that the prediction error baseline signals context changes to improve adaptability. Critically, the neural results support this view, demonstrating multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free learning.One sentence summaryA theoretical, behavioral, computational, and neural account of how the brain resolves the bias-variance tradeoff during reinforcement learning is described.

scholarly journals Parallel model-based and model-free reinforcement learning for card sorting performance

2020 ◽  
Vol 10 (1) ◽  
Author(s):  
Alexander Steinke ◽  
Florian Lange ◽  
Bruno Kopp
Keyword(s):  
Reinforcement Learning ◽  
Cognitive Flexibility ◽  
Theoretical Perspective ◽  
Behavioral Flexibility ◽  
Wisconsin Card Sorting Test ◽  
Card Sorting ◽  
Learning Models ◽  
Model Based ◽  
Model Free ◽  
Reinforcement Learning Models

Abstract The Wisconsin Card Sorting Test (WCST) is considered a gold standard for the assessment of cognitive flexibility. On the WCST, repeating a sorting category following negative feedback is typically treated as indicating reduced cognitive flexibility. Therefore such responses are referred to as ‘perseveration’ errors. Recent research suggests that the propensity for perseveration errors is modulated by response demands: They occur less frequently when their commitment repeats the previously executed response. Here, we propose parallel reinforcement-learning models of card sorting performance, which assume that card sorting performance can be conceptualized as resulting from model-free reinforcement learning at the level of responses that occurs in parallel with model-based reinforcement learning at the categorical level. We compared parallel reinforcement-learning models with purely model-based reinforcement learning, and with the state-of-the-art attentional-updating model. We analyzed data from 375 participants who completed a computerized WCST. Parallel reinforcement-learning models showed best predictive accuracies for the majority of participants. Only parallel reinforcement-learning models accounted for the modulation of perseveration propensity by response demands. In conclusion, parallel reinforcement-learning models provide a new theoretical perspective on card sorting and it offers a suitable framework for discerning individual differences in latent processes that subserve behavioral flexibility.

Bayes factors for reinforcement-learning models of the Iowa gambling task.

Decision ◽  
2016 ◽  
Vol 3 (2) ◽  
pp. 115-131 ◽  
Author(s):  
Helen Steingroever ◽  
Ruud Wetzels ◽  
Eric-Jan Wagenmakers
Keyword(s):  
Reinforcement Learning ◽  
Iowa Gambling Task ◽  
Bayes Factors ◽  
Gambling Task ◽  
Learning Models ◽  
Reinforcement Learning Models

scholarly journals Test-retest reliability of canonical reinforcement learning models

2019 ◽  
Author(s):  
Laura Weidinger ◽  
Andrea Gradassi ◽  
Lucas Molleman ◽  
Wouter van den Bos
Keyword(s):  
Reinforcement Learning ◽  
Learning Models ◽  
Retest Reliability ◽  
Reinforcement Learning Models ◽  
Test Retest Reliability
Sign in / Sign up

Export Citation Format

Share Document