| Talking face video generation aims to automatically synthesize the talking video of the target person by the computer according to a given speech data and a face photo,requiring the speaker’s mouth changes to keep pace with the audio,and facial movement natural and smooth.The technology is widely used in fields such as film production and virtual interaction.However,the mapping relationship between audio and video is affected by many factors such as modal heterogeneity,face authenticity and time continuity.As a result,the visual effects of face videos generated by existing methods still need to be improved.In addition,video generation technology is a double-edged sword.Criminals use fake facial videos to blackmail and mislead public opinion,seriously threatening personal privacy,social stability,and even national security.Therefore,the development of fake video detection technology has also become an urgent need.To sum up,in order to make video generation technology better serve legitimate industries and avoid it being used for illegal purposes,based on the deep neural network and from the perspective of game theory,this paper conducts in-depth research on talking face video generation and detection algorithm with mutual offensive and defensive relationship between them.Based on previous work,corresponding improved models are proposed respectively.The main work is as follows:(1)Aiming at the weak generalization of video generation methods based on 3D model and the poor operability of video generation methods based on 2D image,this paper proposes a talking face video generation method based on multi-dimensional landmarks fusion.Firstly,3D face reconstruction was carried out on the given target image to obtain the 3DMM parameters of the 3D model.Secondly,the long shortterm memory network was used to predict the landmark displacement of the lip from the audio signal,and through a novel temporal residual loss to optimize its temporal synchronization with audio.At the same time,establish a general mapping from audio to facial expression and head posture,predict facial expression and posture parameters and replace original parameters with them.The processed combination parameters are used to drive the 3D face model to obtain 3D face animation,and the animation is mapped to the two-dimensional plane to obtain the landmark displacement of the face.Then,the complete landmark sequence is obtained by fusing all the landmark displacements.Finally,the target image and complete landmark sequence are input into a video synthesis network based on multi-path polarized self-attention and weighted residual to generate the final talking face video.The experimental results show that the proposed method can generate high quality talking face video with lip motion that synchronized with audio and natural head motion,and the performance indexes are better than the existing methods.(2)Aiming at the problems such as low model accuracy,poor generalization ability and weak anti-interference capability of existing forged facial video detection methods,this paper proposes a new forged facial video detection method based on the improved EfficientNetV2 network and multi-feature fusion.Firstly,the key frame is extracted from the face video by using the inter-frame difference to effectively reduce the amount of model calculation.Secondly,the EfficientNetV2 which integrated frequency domain channel attention is used as the backbone network,and build a new attention module,through the use of deep semantic features to predict multiple spatial attention maps,make the network focus on different face regions,and extract RGB features.Then,the texture features are extracted and enhanced by the shallow network to prevent the subtle artifacts from disappearing in the deep network.In addition,Bi LSTM is used to learn the inter-frame inconsistency characteristics of key frames after power spectrum conversion.Finally,all the extracted features are fused and input into the full connection layer,and the detection results of the input video are obtained by using the Softmax function.A large number of experiments show that,compared with the existing detection methods,the proposed method has the best detection performance,not only has higher detection accuracy in cross-data set evaluation,but also has good robustness to common image interference. |
