RGB-D Face Repairing And Recognition With Partial Missing Information

With the rapid development of deep learning,the 2D face recognition has made remarkable progress in the last decade.However,2D face recognition system cannot perform well in complex scenes,such as large pose,occlusion and illumination.3D face data carries extra spatial information and has the invariance of postural and illumination,which improves the accuracy of face recognition to certain extent.However,3D face recognition development is limited by the small scale of 3D face datasets set and the complexity of data pretreatment and massive amount of calculation.RGB-D face recognition combines the advantages of both.While data image acquisition costs are low,it also provides additional facial spatial structure information.However,the current consumer depth sensors will have the phenomenon of missing depth images when collecting data.Therefore,how to repair the missing area in depth image is an important problem.In addition,how to effectively fuse the double modal information of texture and depth in RGB-D data,so as to improve the RGB-D face recognition effect is also the research focus of this thesis.To solve the problem of RGB-D Depth image data loss,a texture-depth generation adversity-network model(color-depth GAN,C-D GAN)is proposed for Depth image restoration.By combining color information,depth information,and edge structure information,a two-stage network was used to optimize the restoration depth and improve the depth map’s effect.Due to the lack of some structural information in the rough restoration depth image obtained by the depth generation network,the restoration image has structural deviation.By introducing the edge structure graph of RGB image as a constraint,the lost structure information is made up to ensure the consistency of the global and local structure of the restored image.At the same time,inspired by the attention mechanism,C-D GAN builds a structurally consistent attention module in the depth repair network considering the patch correlation between regions to be repaired to obtain better depth repair results.The C-D GAN model in this thesis effectively fixes the problem of depth loss in RGB-D face data.To improve RGB-D face recognition accuracy,this thesis proposes a Double Modality Transformer Recognition Network(DMTR-NET)based on Transformer.In this thesis,the RGB-D face recognition network model ensures that texture features and spatial features do not interfere with each other,and through the dual-mode fusion module to make texture features and spatial features mutual fusion,effectively improving the RGB-D face recognition accuracy.Experiments are carried out on multiple RGB-D datasets,the results of the proposed method on multiple test subsets exceed those of the existing benchmark methods.In this thesis,RGB-D face data is used to repair the missing areas in depth images,and a dual-mode fusion recognition network is constructed to further improve the accuracy of face recognition.