Video Super-Resolution Based On3D Residual Network

Video super-resolution reconstructs low-resolution video into high-resolution video,it effectively improves the quality of video and provides better visual experience for human eyes.Video super-resolution is widely used in surveillance,communication,entertainment and other fields.With the development of deep learning,great progress has been made in video super-resolution,but there are still many shortcomings in how to better utilize and fuse the spatio-temporal information between inter-frames and intra-frames.Aiming at this problem,the main work of this paper is as follows:1.Based on residual networks,this paper experiments and compares the performance of three fusion methods for video super resolution:2D early fusion,2D slow fusion and 3D fusion.Considering that 3D residual fusion has good performance but has too many parameters and runs slow,a 3D spatio-temporal seperate residual network which can effectively reduce the parameters and improve the speed is proposed.2.For the motion compensation methods in the video super-resolution task,this paper studies the influence of optical flow method and deformable convolutional method.This paper finds Spynet optical flow can effectively improve the performance of video super-resolution.Experimental results show that Spynet can increase PSNR(Peak signal-to-noise Ratio)by 0.07 dB.3.Considering that it is hard to extract similar image block information with large spatio-temporal spans in videos for convolution layer,a 3-element non-local neural network is proposed.Embedded in the in the shallow layer of the video super-resolution network,the 3-element neural network can enhance the similar image block information extraction with large spans between inter-frames and intra-frames,and make better use of similar block information in video.Experimental results show that the 3-element non-local neural network can improve the super-resolution effect by0.1d B.4.Considering that the adjacent frames has inconsistent contribution to reference frame restoration,this paper proposes a temporal attention mechanism,which enables the network to pay more attention to the video frames that contribute more to reference frame and improves the super-resolution performance.Experimental results show that the temporal attention mechanism can improve the super-resolution effect by 0.15 d B.In addition,this paper also uses progressive up-sampling to replace one-step upsampling,which reduces the learning difficulty of the network and further improves the video super-resolution performance.The final experimental results show that the algorithm in this paper achieves better results than the previous algorithms,and ranks first among the multi-frame video super-resolution algorithms.