An FMRI Study Of Prediction Error Representation And Learning During Decision Making

During decision making, a prediction error (PE) arises if there is a difference between the actual outcome and the expected. There are two kinds of PE, the reward PE and the risk PE. Computational models of reinforcement learning have highlighted the role of PE in learning from outcomes. Neurobiological studies suggest that such learning depends on midbrain dopamine neurons and their striatal and prefrontal targets etc.However, the roles of these brain regions in PE processing in reinforcement learning are still unclear. We proposed such a question in view of the following three aspects:First, both rule uncertainty and outcome uncertainty are important in reinforcement learning. Rule uncertainty refers to the unknown association between two consecutive events. Classical reinforcement learning theories assume that such association could be learned by PE. By contrast, in the case of probabilistic reinforcement, there will be outcome uncertainty due to the probability event. Therefore, there will be a PE after each event, even if our predictions are correct on average and the association (i.e., the rule) never changes. In such cases, PE occurring does not mean that the association has changed, and it would not be useful to update the association. To account for this situation, one can quantify the expectation for the PE, and rule change should only be inferred if the actual PE is larger than expected. Hence, dissociated PE processing in a probabilistic event (i.e., processing for an expected PE that does not mean a changed rule, and we named it PE representation) and in learning (i.e., PE processing for prediction update, and we named it PE learning) in reinforcement learning is evident in human behavior.Second, despite these known behavioral effects, the neural mechanisms that govern these differentiated PE processing are not well understood. Rather than to separate neural activation in different processing, previous studies usually focused on general neural coding of PE and its allignment with behaivoral data. Schonberg et al. reported that the reward PE-related activation in the striatum distinguished learners from non-learners during decision making under ambiguity. However, the bad performance of "non-learners" in that study could be resulted from failed PE learning or PE representation in reinforcement learning. Therefore, we still don’t know for sure about the role of the striatum in PE processing. However, other lines of evidence implicate separate neural systems in PE processing in reinforcement learning. Preuschoff et al. reported neural activation related to risk PE in the anterior insula. However, the task design in their study did not provide an incentive for learning, such that risk predictions need not to be updated. Another fMRI study of decision making under ambiguity found that risk PE involved more brain regions like rostral-anterior cingulate cortex (rACC, in addition to the anterior insula), etc. Concerning both these two results, we assumed that there would be some common neural basis (e.g., the insula) between conditions that requires learning or not suggesting PE representation, and that there would be some extra brain regions (e.g., the rACC) only under conditions that requires learning suggesting PE learning. However, direct evidence of the separated neural system of PE processing is still absent yet. Besides risk PE, the relevant studies about the reward PE are not found, either. Therefore, the neural mechanism of PE learning still awaits formalization.Third, there are two main theories of dopamine’s function from late1990s to the present. The reward prediction error hypothesis (RPEH) has usually taken form of teaching signals or prediction errors in reinforcement learning. By contrast, another theoretical assumption, incentive salience hypothesis (ISH), indicates that dopamine does not actually cause learning about reward, and the coding of learning may be partly illusory because it occurs primarily under confounded conditions that conflate learning with motivation. Therefore, to clarify the specific roles of PE-related brain regions in reinforcement learning may help us understand the role of dopamine system.In brief, the main question in this study centers on the neural basis of PE representation and PE learning.In this study, we adopted the Iowa Gambling Task (IGT) and a probabilistic risk decision task (RDT). Both in IGT and RDT, participants were required to make decisions among four choices for reward. The difference of these two tasks is that whether a participant needs to learn PE or not for advantageous choices. That is, PEs are only represented in RDT, but represented and learned in IGT. Therefore, we supposed a common set of PE-related brain regions between IGT and RDT that may be related to PE representation, and an additional set of PE-related brain regions (IGT> RDT) that may be associated with PE learning. Furthermore, since there might be potentially alternative explanations of the additional brain activation in IGT (e.g., different task complexity), two more analyses were carried out to support the distinct roles of the common set and the additional set of brain regions:first, if a brain region is learning-related, the activation of it should be uncertainty sensitive. So, we analyzed the modulation effect of uncertainty on PE-related brain activation in them; second, if a brain region helps prediction update, there might be some functional connectivity between it and prediction associated brain region. We tested such connectivity with psychophysical interaction analysis.Finally, since most fMRI data are results of correlation but cannot readily demonstrate the necessity, we used high-dimension transcranial direct current stimulation (HD-tDCS) to alter the activation in PE learning-related brain regions. We hypothesized that electric stimulation over them would influence performance in IGT but not in RDT.In this study, we adopted the Iowa Gambling Task (IGT) with unknown reward probabilities and the risk decision making (RDT) with known reward probabilities. In both IGT and RDT, participants need to make decision among multiple choices and choose an advantageous one for reward. The difference is that, in RDT a subject could make a choice without learning from PE, but in IGT the acquisition of advantageous choices need learning from PE. Therefore the essential difference between the two conditions is about PE learning. That is, in RDT, PEs need only to be represented, but in IGT PEs need to be represented and learned. So brain regions both in IGT and RDT are related to PE representation; and brain regions only in IGT but not RDT are related to PE learning.In this fMRI study,41normal participants and another40participants completed IGT and RDT during MRI scanning. Though comparing PE related brain regions between IGT and RDT, we found that,(1) common part of brain regions between IGT and RDT include caudate, anterior insula and r-superior temporal gyrus. Brain regions only activated in IGT include r-anterior cingulate cortex, posterior cingulate cortex and1-middle temporal gyrus;(2) Among common part of brain regions between IGT and RDT, reward PE was related to the caudate, and the risk PE was related to the r-anterior insula/r-superior temporal gyrus, and no brain region was found related to both PEs. Among brain regions only activated in IGT, both PE were related to the rostral anterior cingulate, posterior cingulate and the left middle temporal gyrus, and no brain region was found only related to the reward PE or risk PE. These results suggested that, caudate played a role in the reward PE representation and the r-anterior as well as the r-superior temporal gyrus played a role in the risk PE representation. Specifically, it suggested a parallel representation of the reward PE and the risk PE, and an integrated process of both PE during PE learning.In addition, in order to testify the distinct role of PE representation and learning related brain regions during decision making with explicit and unknown reward probabilities, we analyzed the relationship between the activity of these two kinds of brain regions and reward/risk prediction as well as uncertainty. We found that,(1) only activation of PE learning related brain regions elicited by the reward PE increased along with uncertainty decreased, but activation of PE representation related brain regions didn’t change over uncertainty changing. This suggested that PE elicited activity in PE learning related brain areas gradually increased along with rule mastering level enhanced but that in PE representation related brain areas was basic and stable required;(2) Only PE learning related brain regions (rostral anterior cingulate cortex and posterior cingulate cortex) but not PE representation related brain regions functionally connected with some other brain regions, which overlapped with reward prediction related brain areas.Finally, since most fMRI data are results of correlation but not causality, we hypothesized that transitorily inhibited PE learning related brain regions with high definition-transcranial direct current stimulation (HD-tDCS), performance of subjects in IGT but not in RDT might decreased. The results were consistent with our hypothesis, and this suggested that PE integration related brain areas are critical to but not only correlate with PE in rule learning.In summary, we found brain regions related to the PE representation and learning through comparing the decision making with explicit and unknown probability, in addition, we found parallel representation of the reward PE and the risk PE as well as integrated learning of both PE. Moreover we further tested the distinct function of PE representation and learning during decision making with unknown probability. Finally, we found a critical role of PE learning during decision making with unknown probability.