Content Decoupling Based Method For Real-time Image Deraining

Images contain rich and accurate high-level semantic information,which provides us the main perceptual knowledge from outside,while are the primary sensing information of video surveillance,intelligent traffic,automatic driving and so on.However,under rainy conditions,the visual effect and data quality of the observed image are seriously degraded by rain streaks,which affect the availability and limit the performance of many downstream vision tasks.Image rain streak removal denotes the image restoration technology to eliminate rain degradation from a rainy image and restore a high-quality rain-free image,which becomes the recent hotspot of computer vision.Image deraining can significantly promote the readability and recognizability of rainy images.It is of great practical implication since it can improve the reliability and robustness of the vision perception based intelligent system in rainy scenes while effectively improve the performance of all-weather operations.In recent years,image deraining tasks achieve impressive performance under simple and light rain scenes.However,most existing deraining methods either fail to produce satisfactory restoration results or cost too much computation under the complex and heavy rain conditions or resource-constrained devices,unable to meet the real requirements.The limitations of existing image deraining methods are mainly focused on: residual rain streaks,incomplete background contents and inefficiency for model inference,as follows:(a)the rain streaks have obvious self-similarity,but the existing deraining methods ignore the spatial similarity.The single-scale or simple multi-scale structures have limited modeling ability for characterizing the contexture and crossscale correlation,thus producing deraining results with obvious rain residue.(b)In rainy images,the rain streaks and background texture are highly interwoven,and thus the rain perturbation would further degrade the image content.Most additive modelbased methods barely consider the nonlinear degradation of rain streaks to background textures,resulting in the obvious detail loss and contrast distortion of deraining results.(c)The existing image deraining methods fail to balance the deraining performance and inference speed.The high-accuracy image deraining methods rely on complex architectures and abundant computational resources,unable to meet the real-time application,while the light-weight models are of limited representation ability by the simplified model.To this end,this dissertation carries out the research of real-time image rain streak removal for high precision,high fidelity and high efficiency representation.It focuses on three aspects: multi-scale self-similarity calculation of rain streaks,association learning of rain streak removal and background recovery,decoupled representation of rain layer and background layer as well as the light-weight design,and has achieved the following innovative results:(1)Multi-scale self-similarity progressive fusion network for single image deraining.In view of the insufficient utilization of spatial self-similarity of rain streaks by the existing methods,this dissertation focuses on the exploration and utilization of spatial self-similarity of rain streaks,and introduces the recurrent memory unit to learn the spatial feature relations of rain streaks,so as to promote the modeling accuracy of rain distribution.At the same time,combined with the pyramid architecture network,a multi-scale progressive fusion strategy is proposed to achieve the collaborative representation of cross-scale rain features.Extensive experimental results show that the proposed method achieves significant superiority over other advanced rain removal methods on multiple synthetic and real rainy benchmark datasets,surpassing the best method(Pre Net,CVPR’2019)at that time by 1.33 d B on average on five synthetic datasets.In addition,this dissertation introduces the joint evaluation of image deraining,object detection and segmentation.Compared with the original rainy input,the detection and segmentation accuracy gains the 10% and 15% improvement,respectively.(2)Rain streak removal and background recovery based dynamic association learning network for single image deraining.The existing methods ignore the nonlinear effect of rain perturbation on background textures,resulting in obvious texture loss and contrast distortion in the deraining results.To this end,this dissertation focuses on the joint optimization of rain streak removal and background texture recovery,and proposes a dynamic association learning network,which uses the prior of rain distribution to promote the background texture restoration.The algorithm introduces the efficient feature expression designs,including double branch fusion network and the approximate feature reconstruction at low dimensional space,which greatly improves the inference speed while maintaining the model performance.Extensive experimental results show that the proposed method achieves impressive superiority than other advanced rain removal methods on multiple synthetic and real rainy benchmark datasets.It exceeds the best method(MPRNet,CVPR’2021)at that time by 0.19 db on the Test1200 synthetic dataset,while reducing the inference time,model parameters and computational cost by 47.3%,19.1% and76.8%,respectively.In addition,extensive experimental results on the mainstream image enhancement tasks(including image deraining,image dehazing and low light enhancement)and joint detection tasks show that the proposed DANet model has good universality and robustness.(3)Multi-stage coupling representation between rain layer and background layer for real-time image deraining.The existing methods either fail to produce satisfactory restoration results or cost too much computation.To this end,this dissertation focuses on the multi-stage coupling representation between rain layer and background layer for real-time image deraining,and proposes a decoupled mechanism between the rain streak layer and the background layer to realize the progressive separation.A light-weight framework with the reasonable asymmetric representation and U-shaped coding structure is further designed to achieve the high-efficiency inference,so as to meet the real-time application.Extensive experimental results on multiple synthetic and real rainy benchmark datasets show that the proposed method achieves impressive improvement than the existing lightweight image deraining methods.Meanwhile,it gains 6.45 d B on the Test1200 dataset to the mainstream light-weight model(LPNet,TNNLS’2020)with the approximate inference speed,while achieving the real-time inference by 35 frames per second on the image with resolution of 512 × 512.In summary,this dissertation focuses on scientific problems of calculating the spatial self-similarity of rain streaks,the association learning mode between rain streak removal and background recovery,the decoupling representation of rain layer and background layer as well as the light-weight design in the image deraining task,and creatively solves three technical bottlenecks of residual rain streaks,incomplete background recovery and inefficiency for model inference in the practical application.The research results can effectively promote the image recognizability and readability of rainy scenes,and guarantee the accuracy of the downstream vision tasks,which provides an effective solution for the all-weather operation in video surveillance,automatic driving application,and so on.