| Reward predication error(RPE)is the difference between the expected reward and the actual reward in a specific environment,and the purpose of learning is to minimize this difference.Exploring the dynamic coding characteristics of RPE in the process of learning in animal brain can help us understand the learning mechanism of biology,and inspiring the reinforcement learning algorithm.In this paper,pigeons are used as model animals,and the ventral tegmental area(VTA)related to the RPE is used as the target brain area.First,we designed the experimental paradigm of reinforcement learning based on operant conditioning,and built an experimental platform for pigeons’ learning.Next,from the perspective of neural activity signal,we determined the dynamic change rule of RPE signal in pigeon learning process,and proposed a Q-learning binary classification algorithm with variable learning rate based on this rule.Finally,the effectiveness of our algorithm is verified by using public data sets.This study has a positive effect on the study of animal learning mechanism.The completed work is summarized as follows:(1)An experimental platform for reinforcement learning based on operant conditioning is built.Firstly,we designed an experimental paradigm of reinforcement learning,here,buttons with different light colors are used as cues,and the food given by pecking a specific color key is used as reward feedback.Next,according to the experimental paradigm and experimental purpose,we built a reinforcement learning training device based on operant conditioning,including light stimulation unit,pecking key detection unit,food reward unit.the device can mark the time of each event in the process of experiment through real-time communication with neural signal acquisition equipment and computer.Finally,we used the system to collect the neural signals of 16 channels in the VTA brain area of the pigeon during the reinforcement learning experiment.(2)The dynamic coding characteristics of RPE signal in pigeons’ VTA brain area were analyzed.Firstly,we determined the task related time window of the experimental paradigm based on the behavioral data.Then,we analyzed the Spike fire rate of pigeons in different stages of learning process by using the method of significant difference test,and here we mainly discussed the law of the change of the neural activity intensity of the cue and reward moment with the reinforcement learning process.The results showed that in the process of reinforcement learning in pigeons,the Spike emission characteristic of the VTA brain area encodes RPE signal,and it encodes the RPE signal for high-value and low-value clues differently.For high-value cues,the RPE signal gradually moved forward from reward to cue,and the change was first fast and then slow,while for low-value cues,the RPE signal at the cue remains unchanged.(3)Inspired by the dynamic change of RPE signal in pigeon reinforcement learning,a Q-learning binary classification algorithm with variable learning rate is proposed.Firstly,we used different polynomial parameters to represent different actions,and each label of data can get a Q function.Secondly,the label of samples is determined by comparing the size of the corresponding Q value of each action,and the learning rate which gradually decreases with the number of update iterations is applied to the update of Q function.Finally,we carried out numerical experiments on UCI public data set to test the proposed algorithm.The results show that the proposed algorithm not only has advantages in classification accuracy,but also has better robustness than other algorithms when the proportion of training set and test set change,which further proves the classification performance of the proposed algorithm. |
