scholarly journals A note on the analysis of two-stage task results: how changes in task structure affect what model-free and model-based strategies predict about the effects of reward and transition on the stay probability

2017 ◽  
Author(s):  
Carolina Feher da Silva ◽  
Todd A. Hare
Keyword(s):  
Scientific Progress ◽  
Task Structure ◽  
Decision Task ◽  
Two Stage ◽  
Model Based ◽  
Model Free ◽  
Task Dependence ◽  
Design Choice ◽  
Important Means ◽  
The Impact

AbstractMany studies that aim to detect model-free and model-based influences on behavior employ two-stage behavioral tasks of the type pioneered by Daw and colleagues in 2011. Such studies commonly modify existing two-stage decision paradigms in order to better address a given hypothesis, which is an important means of scientific progress. It is, however, critical to fully appreciate the impact of any modified or novel experimental design features on the expected results. Here, we use two concrete examples to demonstrate that relatively small changes in the two-stage task design can substantially change the pattern of actions taken by model-free and model-based agents as a function of the reward outcomes and transitions on previous trials. In the first, we show that, under specific conditions, purely model-free agents will produce the reward by transition interactions typically thought to characterize model-based behavior on a two-stage task. The second example shows that model-based agents’ behavior is driven by a main effect of transition-type in addition to the canonical reward by transition interaction whenever the reward probabilities of the final states do not sum to one. Together, these examples emphasize the task-dependence of model-free and model-based behavior and highlight the benefits of using computer simulations to determine what pattern of results to expect from both model-free and model-based agents performing a given two-stage decision task in order to design choice paradigms and analysis strategies best suited to the current question.

scholarly journals Credit assignment to state-independent task representations and its relationship with model-based decision making

2019 ◽  
Vol 116 (32) ◽  
pp. 15871-15876 ◽  
Author(s):  
Nitzan Shahar ◽  
Rani Moran ◽  
Tobias U. Hauser ◽  
Rogier A. Kievit ◽  
Daniel McNamee ◽  
...  
Keyword(s):  
Decision Task ◽  
Credit Assignment ◽  
Free System ◽  
Assigned Value ◽  
Model Based ◽  
Independent Task ◽  
Model Free ◽  
Motor Representations ◽  
The Impact ◽  
Task Representations

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

scholarly journals Prefrontal solution to the bias-variance tradeoff during reinforcement learning

2020 ◽  
Author(s):  
Dongjae Kim ◽  
Jaeseung Jeong ◽  
Sang Wan Lee
Keyword(s):  
Adaptive Control ◽  
Reinforcement Learning ◽  
Prediction Error ◽  
Brain Regions ◽  
Decision Task ◽  
Prediction Errors ◽  
Model Based ◽  
Model Free ◽  
Bias Variance ◽  
The Brain

AbstractThe goal of learning is to maximize future rewards by minimizing prediction errors. Evidence have shown that the brain achieves this by combining model-based and model-free learning. However, the prediction error minimization is challenged by a bias-variance tradeoff, which imposes constraints on each strategy’s performance. We provide new theoretical insight into how this tradeoff can be resolved through the adaptive control of model-based and model-free learning. The theory predicts the baseline correction for prediction error reduces the lower bound of the bias–variance error by factoring out irreducible noise. Using a Markov decision task with context changes, we showed behavioral evidence of adaptive control. Model-based behavioral analyses show that the prediction error baseline signals context changes to improve adaptability. Critically, the neural results support this view, demonstrating multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free learning.One sentence summaryA theoretical, behavioral, computational, and neural account of how the brain resolves the bias-variance tradeoff during reinforcement learning is described.

Fault Recognition in the Presence of Error With NURBS-Based Metamodels

2009 ◽  
Author(s):  
Cameron J. Turner
Keyword(s):  
Theoretical Foundation ◽  
Condition Based Maintenance ◽  
On Demand ◽  
Model Based ◽  
Sources Of Error ◽  
Model Free ◽  
Fault Recognition ◽  
Simulated System ◽  
Free Model ◽  
The Impact

Condition-based maintenance (CBM) offers the possibility of replacing the predominant maintenance-as-scheduled paradigm with a maintenance-on-demand paradigm. In all CBM algorithms, faults must first be recognized, then characterized and finally reconciled. Multiple CBM methods have been proposed, including model-free, model-based and metamodel-based methods. However, the signals from real systems are obscured by sources of error. This research examines the impact of error upon a metamodel-based CBM approach using a simulated system to reveal the significance of error in the all-important step of fault recognition. The use of a simulated system allows control of the type and magnitude of both the error and of the fault signals allowing their significance to be evaluated. As a result of this research, a stronger theoretical foundation metamodel-based CBM techniques is established and several promising behaviors are identified.

scholarly journals Humans are primarily model-based learners in the two-stage task

2019 ◽  
Author(s):  
Carolina Feher da Silva ◽  
Todd A. Hare
Keyword(s):  
Simple Model ◽  
Habit Formation ◽  
Learning Processes ◽  
Reward Learning ◽  
Learning Models ◽  
Two Stage ◽  
Model Based ◽  
Model Free ◽  
Task Instructions ◽  
Versus Model

AbstractDistinct model-free and model-based learning processes are thought to drive both typical and dysfunctional behaviours. Data from two-stage decision tasks have seemingly shown that human behaviour is driven by both processes operating in parallel. However, in this study, we show that more detailed task instructions lead participants to make primarily model-based choices that have little, if any, simple model-free influence. We also demonstrate that behaviour in the two-stage task may falsely appear to be driven by a combination of simple model-free and model-based learning if purely model-based agents form inaccurate models of the task because of misconceptions. Furthermore, we report evidence that many participants do misconceive the task in important ways. Overall, we argue that humans formulate a wide variety of learning models. Consequently, the simple dichotomy of model-free versus model-based learning is inadequate to explain behaviour in the two-stage task and connections between reward learning, habit formation, and compulsivity.

scholarly journals Metacontrol of decision-making strategies in human aging

2019 ◽  
Author(s):  
Florian Bolenz ◽  
Wouter Kool ◽  
Andrea M.F. Reiter ◽  
Ben Eppinger
Keyword(s):  
Older Adults ◽  
Decision Making ◽  
Cognitive Abilities ◽  
Reduced Model ◽  
Task Structure ◽  
Costs And Benefits ◽  
Sequential Decision ◽  
Model Based Control ◽  
Model Based ◽  
Model Free

When making decisions, humans employ different strategies which are commonly formalized as model-free and model-based reinforcement learning. While previous research has reported reduced model-based control with aging, it remains unclear whether this is due to limited cognitive capacities or a reduced willingness to engage in an effortful strategy. Moreover, it is not clear how aging affects the metacontrol of decision making, i.e. the dynamic adaptation of decision-making strategies to varying situational demands. To this end, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based control. In contrast to previous research, in this study we applied a task in which model-based control led to higher payoffs in terms of monetary reward. Moreover, we manipulated the costs and benefits associated with model-based control by varying reward magnitude as well as the stability of the task structure. Compared to younger adults, older adults showed reduced reliance on model-based decision making and less adaptation of decision-making strategies to varying costs and benefits of model-based control. Our findings suggest that aging affects the dynamic metacontrol of decision-making strategies and that reduced model-based control in older adults is due to limited cognitive abilities to represent the structure of the task.

scholarly journals Metacontrol of decision-making strategies in human aging

eLife ◽  
2019 ◽  
Vol 8 ◽  
Author(s):  
Florian Bolenz ◽  
Wouter Kool ◽  
Andrea MF Reiter ◽  
Ben Eppinger
Keyword(s):  
Older Adults ◽  
Decision Making ◽  
Cognitive Abilities ◽  
Cross Sectional Study ◽  
Reduced Model ◽  
Task Structure ◽  
Sequential Decision ◽  
Cross Sectional ◽  
Model Based ◽  
Model Free

Humans employ different strategies when making decisions. Previous research has reported reduced reliance on model-based strategies with aging, but it remains unclear whether this is due to cognitive or motivational factors. Moreover, it is not clear how aging affects the metacontrol of decision making, that is the dynamic adaptation of decision-making strategies to varying situational demands. In this cross-sectional study, we tested younger and older adults in a sequential decision-making task that dissociates model-free and model-based strategies. In contrast to previous research, model-based strategies led to higher payoffs. Moreover, we manipulated the costs and benefits of model-based strategies by varying reward magnitude and the stability of the task structure. Compared to younger adults, older adults showed reduced model-based decision making and less adaptation of decision-making strategies. Our findings suggest that aging affects the metacontrol of decision-making strategies and that reduced model-based strategies in older adults are due to limited cognitive abilities.

scholarly journals Explicit knowledge of task structure is the primary determinant of human model-based action

2020 ◽  
Author(s):  
Pedro Castro-Rodrigues ◽  
Thomas Akam ◽  
Ivar Snorrason ◽  
Marta Camacho ◽  
Vitor Paixao ◽  
...  
Keyword(s):  
Explicit Knowledge ◽  
Human Model ◽  
Task Structure ◽  
Obsessive Compulsive ◽  
Model Based Control ◽  
Structure Information ◽  
Model Based ◽  
Model Free ◽  
Before And After ◽  
Novel Variant

Explicit information obtained through instruction profoundly shapes human choice behaviour. However, this has been studied in computationally simple tasks, and it is unknown how model-based and model-free systems, respectively generating goal-directed and habitual actions, are affected by the absence or presence of instructions. We assessed behaviour in a novel variant of a computationally more complex decision-making task, before and after providing information about task structure, both in healthy volunteers and individuals suffering from obsessive-compulsive (OCD) or other disorders. Initial behaviour was model-free, with rewards directly reinforcing preceding actions. Model-based control, employing predictions of states resulting from each action, emerged with experience in a minority of subjects, and less in OCD. Providing task structure information strongly increased model-based control, similarly across all groups. Thus, explicit task structural knowledge determines human use of model-based reinforcement learning, and is most readily acquired from instruction rather than experience .

scholarly journals Identifying Model-Based and Model-Free Patterns in Behavior on Multi-Step Tasks

2016 ◽  
Author(s):  
Kevin J. Miller ◽  
Carlos D. Brody ◽  
Matthew M. Botvinick
Keyword(s):  
Model Fitting ◽  
Decision Task ◽  
Sequential Decision ◽  
Agent Based Model ◽  
Agent Based ◽  
Model Based ◽  
Model Free ◽  
Analysis Technique ◽  
Analyze Data

Recent years have seen a surge of research into the neuroscience of planning. Much of this work has taken advantage of a two-step sequential decision task developed by Daw et al. (2011), which gives the ability to diagnose whether or not subjects’ behavior is the result of planning. Here, we present simulations which suggest that the techniques most commonly used to analyze data from this task may be confounded in important ways. We introduce a new analysis technique, which suffers from fewer of these issues. This technique also presents a richer view of behavior, making it useful for characterizing patterns in behavior in a theory-neutral manner. This allows it to provide an important check on the assumptions of more theory-driven analysis such as agent-based model-fitting.

scholarly journals Modeling and prediction of pressure injury in hospitalized patients using artificial intelligence

2021 ◽  
Vol 21 (1) ◽  
Author(s):  
Christine Anderson ◽  
Zerihun Bekele ◽  
Yongkai Qiu ◽  
Dana Tschannen ◽  
Ivo D. Dinov
Keyword(s):  
Artificial Intelligence ◽  
At Risk ◽  
Hospitalized Patients ◽  
Surgical Services ◽  
Model Based ◽  
Model Free ◽  
Healthcare Needs ◽  
Patients At Risk ◽  
Hospital Acquired ◽  
The Impact

Abstract Background Hospital-acquired pressure injuries (PIs) induce significant patient suffering, inflate healthcare costs, and increase clinical co-morbidities. PIs are mostly due to bed-immobility, sensory impairment, bed positioning, and length of hospital stay. In this study, we use electronic health records and administrative data to examine the contributing factors to PI development using artificial intelligence (AI). Methods We used advanced data science techniques to first preprocess the data and then train machine learning classifiers to predict the probability of developing PIs. The AI training was based on large, incongruent, incomplete, heterogeneous, and time-varying data of hospitalized patients. Both model-based statistical methods and model-free AI strategies were used to forecast PI outcomes and determine the salient features that are highly predictive of the outcomes. Results Our findings reveal that PI prediction by model-free techniques outperform model-based forecasts. The performance of all AI methods is improved by rebalancing the training data and by including the Braden in the model learning phase. Compared to neural networks and linear modeling, with and without rebalancing or using Braden scores, Random forest consistently generated the optimal PI forecasts. Conclusions AI techniques show promise to automatically identify patients at risk for hospital acquired PIs in different surgical services. Our PI prediction model provide a first generation of AI guidance to prescreen patients at risk for developing PIs. Clinical impact This study provides a foundation for designing, implementing, and assessing novel interventions addressing specific healthcare needs. Specifically, this approach allows examining the impact of various dynamic, personalized, and clinical-environment effects on PI prevention for hospital patients receiving care from various surgical services.

Sign in / Sign up

Export Citation Format

Share Document