Research On Deep Spiking Reinforcement Learning Methods Based On Surrogate Gradient Strategy

Spiking Neural Networks(SNN)have been widely studied by researchers in recent years due to their high biological plausibility and low-power potential on smart chips,and are considered as the next generation of Artificial Neural Networks(ANN).Unlike traditional ANN that uses continuous values to characterize information,SNN uses binary-valued discrete spike trains to characterize information.At present,while Deep Reinforcement Learning research has made great progress,Deep Spiking Reinforcement Learning combining SNN and Reinforcement Learning algorithms is still in its infancy.All existing Deep Spiking Reinforcement Learning methods use SNN conversion methods to transform pre-trained traditional ANN into SNN with corresponding structure to achieve indirect training of SNN.This leads to strong limitations of existing methods,for example,the existing methods have a strong reliance on pre-trained traditional ANN,and SNN conversion methods usually require a coding time window of at least hundreds of coding time steps to achieve sufficient conversion accuracy,which prevents the existing methods from taking full advantage of the high energy efficiency of SNN.In this thesis,in order to solve the above problems,a Deep Spiking Reinforcement Learning framework based on direct training method is proposed for the first time.The framework achieves direct training of SNN models based on Surrogate Gradient Strategy,and the framework can achieve high accuracy while shortening the coding time window by one order of magnitude,giving full play to the high energy efficiency advantage of SNN.This thesis introduces Distributional Reinforcement Learning theory based on the framework,extending it from the field of traditional Deep Reinforcement Learning to the field of Distributional Reinforcement Learning,and proposes a Distributional Spiking Reinforcement Learning algorithm.Finally,this thesis conducts sufficient experiments for the proposed framework and Distributional Spiking Reinforcement Learning algorithm,and the experimental results demonstrate the effectiveness of the framework in both the field of traditional Deep Reinforcement Learning and Distributional Reinforcement Learning.The main work of this thesis is as follows:(1)A Directly-Trained Deep Spiking Reinforcement Learning framework based on Q-learning,Leaky Integrate-and-Fire model and Surrogate Gradient Strategy is proposed,which includes a Deep Spiking Reinforcement Learning model based on a direct training method and a direct training method oriented to Deep Spiking Q-Learning.Finally,a theoretical analysis of the framework is presented.(2)A Distributional Spiking Reinforcement Learning algorithm based on the framework,Distributional Bellman Equation and Quantile Regression is proposed,extended the framework from the field of traditional Deep Reinforcement Learning to the field of Distributional Reinforcement Learning.(3)Adequate comparison experiments are conducted in the Atari game task for the Deep Spiking Reinforcement Learning framework adapted to different traditional Deep Reinforcement Learning algorithms and Distributional Spiking Reinforcement Learning algorithm,respectively.The experimental results demonstrate the effectiveness and advantages of the framework in both the field of traditional Deep Reinforcement Learning and Distributional Reinforcement Learning.