Research On Face Image Segmentation And Completion Algorithms

With the continuous development of convolutional neural networks,the research and engineering applications of face-related vision tasks have made significant progress,such as face recognition technology widely used in social and financial fields.Recently,the scale of human-based streaming media has proliferated,including video conferences,short videos,and online lectures.It is becoming more critical for a more intelligent understanding and editing of 2D faces.This dissertation focuses on face understanding and editing from the perspective of face parts.This dissertation proposes two basic face segmentation algorithms and a facial mouth completion algorithm for real-time facial mask removal in video conferences.Face segmentation computes pixel-wise label maps for different semantic components(e.g.,hair,mouth,eyes)from face images.Inspired by the human physiological visual system,this dissertation proposes a novel RoI Tanh-warping image transformation operation that combines the central vision and the peripheral vision.It addresses the dilemma between a limited-sized RoI for focusing and an unpredictable area of surrounding context for peripheral information.It uses a hierarchical local-based method for inner facial components and global processes for outer facial parts.Experiments on two existing benchmarks,including LFW-PL and HELEN,demonstrate that our algorithm surpasses state-of-the-art methods.This dissertation proposes a hybrid convolutional neural network FS-ROI-NSM based on edge attention to address the prediction error problem of facial part edges and regions far from the face center.It adds a branch to improve edge hair prediction accuracy.In FS-ROI-NSM,the RoI Tanh-warping branch and the NSM branch are integrated to deal with relatively fixed internal areas of the face and unknown regions.FS-ROI-NSM has better segmentation performance than FS-ROI on the LaPa dataset.More results show that FS-ROI-NSM can deal with face segmentation in various scenarios.In terms of face completion,this dissertation proposes a novel mask removal algorithm MFRA,which can complete the masked area of the face in real-time.In video conferencing,MFRA allows participants to see others as if they were not wearing a mask.MFRA maps the audio feature to 3DMM expression coefficients to guide the generation of various mouth shapes utilizing multi-modal information and adopt the attribute and identity decoupled generation network to generate the mouth appearance consistent with the reference image.Experiments have verified that the system has a good mask removal effect and can achieve a better video conference experience.Besides,through lightweight network design and engineering optimization,MFRA can run in quasi-real-time.