Research On Key Technologies Of Intelligent Image Signal Processor

Image sensors have been widely employed in security monitoring,medical imaging,automatic driving,virtual reality,intelligent control,industrial inspection,remote sensing observation and other fields.The image signal processor(ISP)is responsible for improving the signal-to-noise ratio(SNR)of the image sensor and restoring the image signal.The performance of ISP determines the imaging quality of the imaging system to a large extent.With the extension of imaging scenarios,traditional ISPs are struggling to meet the requirements for improving the SNR in high resolution,strong noise and backlight conditions.Therefore,intelligent ISP with strong scene adaptability and high processing accuracy has become a research hotspot.In order to realize the intelligent ISP,it is necessary to simultaneously carry out the algorithm research and hardware computing system design.We should constantly enhance the accuracy of the algorithm and improve the energy efficiency of hardware systems,thus supporting the deployment of intelligent image processing algorithms.In the ISP pipelines,noise suppression is the pre-sequence step to improve the image SNR,and also directly affects the accuracy of post-processing tasks such as demosaic,contrast enhancement,and spatial resolution reconstruction,and has become the key technology to improve ISP performance.In view of this,this paper focuses on the image noise suppression task.We deeply analyze the noise characteristics of visible,infrared and remote sensing imaging systems.Then utilizing the algorithm-hardware co-design method to design the lightweight intelligent image suppression algorithm and its energy-efficient hardware accelerator.The main research contents and achievements are as follows:For the noise suppression problem of visible sensors,a lightweight deformable convolution based noise reduction method is proposed.The kernel deformation mechanism is utilized to realize adaptive sampling and improve the model feature extraction ability.The lightweight deformable convolution unit is built with the characteristics analysis of the deformable convolution in the image noise suppression task,which employs an offset sharing strategy to solve the irregular access problem and reduce about 94% computation load in deformable parameter generation.At the same time,the deformation grouping strategy is also proposed,which further reduces 75% computational complexity of the hardware system and enhances the adaptability of kernel deformation.Experimental results show that our method achieves a balance between noise suppression and image detail preservation,compared with the classic DJDD model achieves 5 d B peak signal-to-noise ratio(PSNR)improvement.Aiming at the noise suppression problem of the infrared sensors,the mixed spatial-channel attention mechanism is proposed to separate the aliased infrared noise,which avoids the image detail loss caused by mixed noise conditions.Based on the analysis of the gradient characteristics of infrared non-uniformity noise,a lightweight infrared noise suppression method is proposed,which utilizes haar wavelet decomposition to excavate the complementary characteristics for better detail preservation and significantly reduces about75% computation for intermediate features.Meanwhile,we also adopt the lightweight depthwise convolution unit to further reduce the computational complexity of our model.Experimental results on NVIDIA GTX1080 Ti GPU show that the proposed method consumes 5.21 ms for 512×512 sized images which is 89.58% faster than ICSRN methods,and the structural similarity also increased by 0.0475 on average.In order to solve the noise suppression problem of the remote sensing imaging system,we proposed the spatial-spectral consistent information fusion method to remove the stripe noise.By analyzing the noise characteristics in the spatial domain and mining the details information in the spectral domain,a selective memory unit is constructed.The recurrent convolution mechanism in the selective memory unit significantly improves the consistency of information extraction ability,and benefits high-precision noise separation and details reconstruction.In addition,the recurrent convolution mechanism also provides the weights sharing strategy to reduce the model size to 47 KB,which significantly simplifies the memory system.At the same time,the recurrent fusion unit is utilized to fuse spatial-spectral information according to spectral correlation.The experimental results show that this method achieves 3d B PSNR improvement with only 13.75% parameter size compared with advanced De Net noise suppression algorithms.Meanwhile,our work generates better results in image texture reconstruction and spectral correlation maintenance.Based on the lightweight noise suppression algorithm,we carried out the design of an intelligent hardware accelerator.Firstly,a set of dedicated computational units are designed to be compatible with various neural operators required for noise suppression models.Then the dedicated computing arrays and memory access units are proposed to improve the energy efficiency of hardware systems.In addition,the memory units are configured to support flexible intermediate data staging and improve compatibility for various models.Experimental results on FPGA show that in the visible and infrared noise suppression tasks,the inference speed of high-definition visible light images with 1920×1080 resolution reaches 32 FPS,and achieves 62 FPS for mainstream infrared images with 640×512resolution,which meets the real-time processing requirements of imaging systems.In the remote sensing noise suppression task based on the RNN structure,the peak energy efficiency of the system reaches 46.26 FPS/W,which is 308 times that of the NVIDIA GTX1080 Ti GPU.Compared with the existing FPGA-based image processor,the intelligent hardware accelerator designed in this paper shows great advantages in terms of flexibility and compatibility.It is compatible with mainstream neural operation operators including standard convolution,depth-wise convolution,point-wise convolution,and deformable convolution,and supports the calculation of CNN and RNN models.The intelligent hardware accelerator provides an excellent development platform for the design of intelligent image processing algorithms and effectively promotes the research progress of ISP,which has high academic and practical value.