Long-Term Reward Prediction in TD Models of the Dopamine System

This article addresses the relationship between long-term reward predictions and slow-timescale neural activity in temporal difference (TD) models of the dopamine system. Such models attempt to explain how the activity of dopamine (DA) neurons relates to errors in the prediction of future rewards. Previous models have been mostly restricted to short-term predictions of rewards expected during a single, somewhat artificially defined trial. Also, the models focused exclusively on the phasic pause-and-burst activity of primate DA neurons; the neurons' slower, tonic background activity was assumed to be constant. This has led to difficulty in explaining the results of neurochemical experiments that measure indications of DA release on a slow timescale, results that seem at first glance inconsistent with a reward prediction model. In this article, we investigate a TD model of DA activity modified so as to enable it to make longer-term predictions about rewards expected far in the future. We show that these predictions manifest themselves as slow changes in the baseline error signal, which we associate with tonic DA activity. Using this model, we make new predictions about the behavior of the DA system in a number of experimental situations. Some of these predictions suggest new computational explanations for previously puzzling data, such as indications from microdialysis studies of elevated DA activity triggered by aversive events.

Effects of dopamine on postural control in parkinsonian subjects: scaling, set, and tone

1. This study investigates the effects of parkinsonism and of dopamine replacement therapy (levodopa) on scaling the magnitude of automatic postural responses based on sensory feedback and on predictive central set. Surface reactive torques and electromyographic (EMG) activity in response to backward surface translations were compared in patients with parkinsonism ON and OFF levodopa and in elderly control subjects. Correlations between the earliest postural responses [initial rate of change of torque and integrated EMG (IEMG)] and translation velocity provided a measure of postural magnitude scaling using somatosensory feedback. Correlations of responses with expected translation amplitude provided a measure of scaling dependent on predictive central set because the responses preceded amplitude completion. 2. Parkinsonian EMG responses in six leg and trunk muscles were not later than in elderly control subjects. In fact, quadriceps antagonist latencies were earlier than normal, resulting in coactivation at the knee not present in control subjects. EMG activation was fragmented, with short burst durations and high tonic levels that often returned to baseline with multiple bursts. In addition, parkinsonian responses showed smaller-than-normal agonist extensor bursts and larger-than-normal activation in tibialis and rectus femorus antagonist flexors. 3. Although parkinsonian subjects scaled postural responses to both displacement velocities and amplitudes, their torque response were smaller than those of elderly controls, especially in response to the largest displacement amplitudes. The gain (slope) of postural response magnitude scaling to displacement velocity was similar for parkinsonian and control subjects, although parkinsonian subjects had smaller torques. Parkinsonian subjects were also able to use prediction to scale responses to small expected displacement amplitudes, but many patients did not generate the larger plantarflexion torques required at larger displacement amplitudes. Reduced torque at large amplitudes was associated with less agonist gastrocnemius IEMG, increased tibialis antagonist burst responses, and increased tibialis tonic background activity. 4. Levodopa further reduced the already low magnitude of initial torque and IEMG responses to displacement velocities and amplitudes in parkinsonian patients. The ability to scale postural responses to velocity feedback was not affected by levodopa, but the ability to scale responses to large displacement amplitudes based on central set was worsened by levodopa. Levodopa also significantly reduced the tonic, background levels of EMG, particularly the distal gastrocnemius and tibialis activity. 5. High baseline muscle tone was apparent in parkinsonian subjects from their high background EMG activity in quiet stance, especially in tibialis and quadriceps, and the slow initial velocity of center of mass falling in response to displacements. By reducing tone, levodopa reduced passive stiffness to perturbations without increasing EMG burst magnitudes, resulting in less resistance to external displacements and thus faster center of body mass (COM) displacements. 6. The biggest postural deficit in parkinsonian subjects was not in response latency, pattern, or reactive or predictive scaling of response magnitude, but in quickly generating an adequate level of postural force. Dopamine improved tonic background postural tone but further weakened automatic postural responses to external displacements. Thus the basal ganglia may participate in postural control by regulating appropriate levels of background postural tone and by enabling adequate force generation for resisting external displacements.

Restoration of extensor excitability in the acute spinal cat by the 5-HT2 agonist DOI

1. The decerebrate cat preparation with an intact spinal cord is characterized by a high degree of excitability in extensor motoneuron pools, which is eliminated by acute spinalization. Subtype-specific agonists for serotonin (5-HT) were investigated in terms of their effectiveness in restoring the extensor excitability following spinalization. 2. Our hypothesis was that 5-HT2 receptors have the primary role in enhancement of extensor reflex excitability, whereas 5-HT1A and 5-HT1B/D receptors are relatively unimportant. Reflex excitability was assessed from the tonic levels of force and electromyographic (EMG) output from the ankle extensors medial gastrocnemius (MG) and soleus (SOL), and from the reflex forces in both these muscles generated by ramp-and-hold stretches of MG. 3. Before spinal transection, MG and SOL usually exhibited a small amount of tonic background EMG activity and force output. Ramp-and-hold stretch of MG generated a large-amplitude reflex response. Spinal transection at the level of T10 virtually abolished tonic background activity in both extensors and greatly attenuated the MG stretch reflex. Ventral topical application of the selective 5-HT2A/2C agonist (+-)-1-(2,5-dimethoxy-4-iodophenyl)-2-amino-propane hydrochloride (DOI) restored the amplitude of the MG stretch reflex in a dose-dependent fashion. However, a considerable portion of the DOI-mediated restoration of MG stretch reflex force was due to elevation of tonic background force levels above previous intact cord levels. 4. The DOI-induced increase in extensor tonic background excitability and facilitation of MG stretch reflex were reversed by ventral topical administration of the selective 5-HT2 antagonist ketanserin. No increase in extensor excitability was observed in spinalized preparations after administration of either the 5-HT1A agonist (+-)-8-hydroxy-dipropylaminotetralin hydrobromide or the 5-HT1B/1D agonist 7-trifluoromethyl-4-(4 methyl-1-piperazinyl)-pyrrolo[1,2- a]quinoxaline maleate. These data strongly suggest that the DOI-induced facilitation of extensor stretch reflex and tonic activity in spinalized preparations is mediated through an action on spinal 5-HT2 receptors. 5. One important difference between the actions of DOI in spinalized versus intact states was that the DOI-induced tonic and reflex forces in the spinalized state were subject to irregular oscillations. In contrast, DOI did not noticeably affect the smoothness of reflex force generation in the intact state. This discrepancy was probably due to the effects of clasp knife inhibition from muscular free nerve endings, which have potent reflex actions in the spinalized but not intact states. Thus DOI elevated excitability levels but did not alter the effects of spinalization on stretch reflex patterns.