Goal-Directed Decision Making

Computations and Neural Circuits

Richard W. Morris, Aaron Bornstein, Amitai Shenhav

• Publisher: Academic Press
• Year: 2018
• Print ISBN: 9780128120989
• eBook ISBN: 9780128120996
• Topic: Psychology
• Subtopic: Neuroscience
• Index: Psychology

Goal-Directed Decision Making: Computations and Neural Circuits examines the role of goal-directed choice. It begins with an examination of the computations performed by associated circuits, but then moves on to in-depth examinations on how goal-directed learning interacts with other forms of choice and response selection. This is the only book that embraces the multidisciplinary nature of this area of decision-making, integrating our knowledge of goal-directed decision-making from basic, computational, clinical, and ethology research into a single resource that is invaluable for neuroscientists, psychologists and computer scientists alike.The book presents discussions on the broader field of decision-making and how it has expanded to incorporate ideas related to flexible behaviors, such as cognitive control, economic choice, and Bayesian inference, as well as the influences that motivation, context and cues have on behavior and decision-making.- Details the neural circuits functionally involved in goal-directed decision-making and the computations these circuits perform- Discusses changes in goal-directed decision-making spurred by development and disorders, and within real-world applications, including social contexts and addiction- Synthesizes neuroscience, psychology and computer science research to offer a unique perspective on the central and emerging issues in goal-directed decision-making

Chapter 1

Actions and Habits

Psychological Issues in Dual-System Theory

Anthony Dickinson¹, and Omar D. Pérez^{2 1}Department of Psychology, University of Cambridge, Cambridge, United Kingdom ²Division of the Humanities and Social Sciences, California Institute of Technology, California, United States

Abstract

Instrumental behavior is controlled by two parallel systems that summate in the control of behavior. One system controls goal-directed action that is based on the interaction of a desire for the outcome of the action (desire criterion) with a belief that the action causes the outcome (belief criterion) through a process of practical inference. The desire criterion is assessed by outcome revaluation procedures, whereas the belief criterion requires that the revaluation effect be mediated by the instrumental, action–outcome contingency. The second system controls habitual responding, which fails to meet the joint requirements of the belief and desire criteria. Rather habits are elicited either directly by the training context or indirectly by a sensory expectation of the reinforcer of the habit and motivated by Pavlovian conditioned motivational states. Simulation of a dual-system theory, in which the instrumental beliefs of the goal-direct system are based on the experienced correlation between the action and outcome rates, explains how the sensitivity of performance to outcome revaluation varies with the reinforcement schedule, the amount of training, and choice rather than single-response training.

Keywords

Goal-directed action; Habits; Instrumental learning; Outcome revaluation; Pavlovian–instrumental interactions

Over a quarter of a century ago, Heyes and Dickinson (1990) offered two behavioral criteria for the attribution of intentionality to animal action that have subsequently served to characterize goal-directed behavior (de Wit & Dickinson, 2009). These criteria are rooted in the folk psychology of action. If asked to explain why a hungry rat is pressing a lever for food pellets, the folk psychologist tells us that the rat desires access to the food pellets and believes that pressing the lever will yield access. This account is deceptively simple in that it has two explicit mental entities, the belief and desire, and an implicit process for deploying these entities in the control of behavior. In spite of this apparent simplicity, the belief–desire account has psychologically important features that motivate our conception of goal-directed action.

A philosopher of mind may well point out that beliefs and desires are particular types of mental entities, propositional attitudes, which have two important features. First, the content of the belief or desire, such as believing that “lever pressing causes access to food pellets” or desiring that “there is access to food pellets,” is a representation of an event, state, or relationship with a propositional-like structure. Second, this propositional content stands in a relationship, or attitude, to the event or state of affairs that is represented. A belief represents a supposed state of affairs in the world and therefore has a world-to-representation fit in that it can be either true or false of the world (Searle, 1983). By contrast, a desire has a representation-to-world fit in that its content represents a state of affairs that is currently not true but that the agent wishes it to be so. Therefore, a desire's fit to the world is one of fulfillments in that the content of the desire can either be fulfilled or unfulfilled.

The third, and often implicit, component of the belief–desire account is a process of practical inference that takes the belief and desire as it arguments to yield an intention to act. As our aim was to marshal beliefs and desires in a psychological account, rather than a philosophical analysis of action, we choose to present the content and the practical inference in a programming language, PROLOG, which was designed, at least in part, to simulate cognitive processes and is reasonably transparent with respect to content. So a minimal program to generate lever pressing for food pellets might take the following form:

cause(lever-press,access(food-pellet),g).	—belief
access(food-pellet,δ).	—desire
perform(A,g∗δ):-cause(A,access(O),g),access(O,δ).	—practical
	inference

This little program is similar to that offered in Heyes and Dickinson (1990) except for the addition of the parameters g and δ, which serve to quantify the believed strength of the causal relationship between lever pressing and access to the food pellets and the strength of the desire for this goal, respectively. Therefore, g represents the believed reliability or rate with which lever pressing will cause access to the food pellets. If we ran this little program to determine which action this impoverished agent intends to perform, it would return an intention to lever-press in the form perform(lever-press) with the will to execute the intention determined by the product of g and δ. If either of these parameters is zero, there will be a lack of will to execute the intention. Henceforth, we shall ignore the quantification of beliefs and desire except where necessary.

This psychological account of action is both causal and rational. It is causal in the sense that it is the interaction of the content of the belief and desire in the process of practical inference that determines the content of the intention, and hence the particular goal-directed action performed. Second, and importantly, the practical inference process yields a rational action. If the belief cause(lever-press,access(food-pellet)) is true and the intention perform(A) is executed then, other things being equal, the desire access(food-pellet) must, of necessity, be fulfilled. The psychological rationality of lever pressing within the present context may appear obvious, and possibly trivial, but, as we shall discuss below, the issue of whether representational content can cause responses that are nonrational with respect to that content is a matter of dispute.

This belief–desire account led Heyes and Dickinson to offer two behavioral criteria for determining whether a particular behavior is an intentional or goal-directed action: the desire (goal) and belief (instrumental) criteria. We shall discuss each in turn.

Desire criterion

A straightforward prediction of the belief–desire account is that, if following lever-press training under the desire for access to food pellets, this desire is reduced, the animal's propensity to press the lever should immediately decrease without any further training. Reducing the desire for access to the food pellets by setting the δ parameter in the little PROLOG program to zero yields an intention to lever-press but without any will to execute it. When Adams (1980) first attempted to assess the status of lever pressing for food pellets in Dickinson's lab by removing his rats' desire for these pellets, he could find no evidence that lever pressing was goal-directed. Having trained the rats to press a lever for access to the food pellets, he then reduced the desire for these pellets by conditioning a food aversion to the pellets in the absence of an opportunity to lever-press. When once again given the opportunity to press the lever (in the absence of the food pellets), the rate of pressing was totally unaffected by whether or not the food pellets were desired at the time.

We were surprised that our rats were insensitive to outcome devaluation, given that many years before Tolman and Gleitman (1949) had convincingly demonstrated that devaluing one of the goal boxes in an E-maze by associating it with electric shocks induced an immediate reluctance by their rats to take the turn leading to the devalued goal. Therefore, we suspected that it was the type of action and training that might be critical in determining whether performance was sensitive to removal of a desire through goal or outcome devaluation.

Adams (1980) trained his rats to press the lever on a variable interval schedule, which models a resource that depletes and then regenerates with time, such as nectar, in that the schedule specifies the average time interval that has to elapse before the next reward becomes available for collection. By contrast, foraging in a nondepleting source is modeled by ratio schedules, which specifies the probability with which each action yields an outcome. Therefore, there is a more direct causal connection between the action and the outcome under a ratio contingency in that it does not involve an additional causal process, outcome regeneration with time. When we switched to training on a variable ratio schedule, we found that lever pressing for food by hungry rats could be goal-directed, at least by the desire criterion. Our rats pressed the lever at a reduced rate following the removal of the desire for the food pellets by aversion conditioning when tested extinction so that the devalued pellets were not presented (Adams & Dickinson, 1981). Subsequently, we established that the type of training schedule is critical in determining whether or not an action is goal-directed. Ratio-trained lever pressing for food is more sensitive to outcome devaluation than interval-trained responding even when...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• Contributors
• Preface
• Chapter 1. Actions and Habits: Psychological Issues in Dual-System Theory
• Chapter 2. Instrumental Divergence and Goal-Directed Choice
• Chapter 3. The Temporal Dynamics of Reward-Based Goal-Directed Decision-Making
• Chapter 4. Case-Based Decision Neuroscience: Economic Judgment by Similarity
• Chapter 5. Learning Structures Through Reinforcement
• Chapter 6. Goal-Directed Sequences in the Hippocampus
• Chapter 7. Competition and Cooperation Between Multiple Reinforcement Learning Systems
• Chapter 8. Cortical Determinants of Goal-Directed Behavior
• Chapter 9. Distinct Functional Microcircuits in the Nucleus Accumbens Underlying Goal-Directed Decision-Making
• Chapter 10. Studying Integrative Processing and Prospected Plasticity in Cholinergic Interneurons: The Importance of Pavlovian–Instrumental Transfer
• Chapter 11. Does the Dopaminergic Error Signal Act Like a Cached-Value Prediction Error?
• Chapter 12. A State Representation for Reinforcement Learning and Decision-Making in the Orbitofrontal Cortex
• Chapter 13. The Development of Goal-Directed Decision-Making
• Chapter 14. Social Learning: Emotions Aid in Optimizing Goal-Directed Social Behavior
• Chapter 15. The Balance Between Goal-Directed and Habitual Action Control in Disorders of Compulsivity
• Chapter 16. Drug Addiction: Augmented Habit Learning or Failure of Goal-Directed Control?
• Chapter 17. Goal-Directed Deficits in Schizophrenia
• Chapter 18. Realigning Models of Habitual and Goal-Directed Decision-Making
• Chapter 19. The Motivation of Action and the Origins of Reward
• Index