| With the rapid development of Internet technology and the popularity of various video recording devices,network video business has shown explosive growth,and video has become an important communication medium and information carrier indispensable in people’s daily life.Video data contains a huge amount of available information and stores much more information than traditional media types,and its classification results are of great significance for the understanding and analysis of video content.The analysis research based on video data has significant theoretical value and broad application prospects.Although supervised learning has been successfully applied to various video classification tasks,it still relies on large-scale video annotations and suffers from issues such as data bias,poor model generalization,and adversarial attacks.In contrast,multimodal video contrastive learning does not require manual annotation and instead utilizes auxiliary tasks to mine supervised information from large-scale unlabeled data to train the network,thereby learning valuable representations for downstream tasks.Around the problem of video classification,we have completed the transition from single-modal to multi-modal,from supervised learning to self-supervised contrastive learning,and carried out the following three research works,and developed a system that integrates multi-modal information for identity recognition.The main research content and contributions are as follows:(1)A Multi-level Perception Feature Aggregation Network(MPFAN)is proposed for video face recognition.Compared with traditional feature aggregation methods,this approach does not require manually designed aggregation coefficients.Instead,permutation-invariant U-Net network is utilized to automatically generate aggregated features.Moreover,the aggregated features are further learned to enhance their robustness.Experimental results show that this algorithm outperforms state-of-the-art methods on several established video-based face recognition benchmarks.(2)A Network for Multi-modal Self-Supervised Learning from Video and Audio(VANet)is proposed.This work takes advantage of the natural synchronization and correlation of video and audio to establish an auxiliary task,and introduce a contrastive loss for joint embedding learning of the two input modalities,which is improved to enhance the proximity of semantically relevant samples and meet the special needs of multi-modal learning.In addition,we identify and remove pseudo-negative samples from the negative set.Our method achieves competitive results in video action recognition tasks and demonstrates that joint multimodal learning is superior to single modality.(3)A Transformer for Multi-modal Multi-label Self-Supervised Learning(MMT)is proposed.We consider two different embedding methods,using pre-trained models of images for initializing video models,and aligning video-audio modality and video-text modality separately in the feature space according to the semantic granularity of different modalities.Experimental results show that a pure attention-based model with multimodal video inputs,large-scale self-supervised pre-training,can alleviate the data burden of the Transformer architecture and have promising directions in outperforming convolutional neural networks in downstream tasks.(4)Building upon previous research,a identity recognition system has been developed based on two video classification algorithms: the Multi-level Perception Feature Aggregation Network(MPFAN)and the Network for Multi-modal Self-Supervised Learning from Video and Audio(VANet).By fusing audio-visual modality information,the accuracy of identity recognition is improved,enabling the completion of identity recognition tasks in complex scenarios.The system can be applied in access control,attendance,online identity verification,and other related fields. |
