| In the context of 5G era,video data has shown explosive growth,providing people with rich and colorful visual information and entertainment experience.However,how to utilize and retrieve videos efficiently has become an urgent problem to be solved.Video moment retrieval task is a solution to this problem.Its goal is to locate a video clip that best matches the semantics of the query description in an untrimmed video according to the natural language query description input by the user,and return its start time point and end time point.At the same time,this task can promote the development of other related fields,such as video summarization,video question answering and video understanding.Although the task of video moment retrieval has high application value and research significance,there are still many challenges to obtain good retrieval results.One of the main challenges is how to effectively process and fuse information between different modalities(i.e.,text and video)and achieve precise semantic alignment.Most of the existing methods deal with each modality independently and extract rough features,resulting in information loss.Or the interaction or correlation between modalities can only be partially explored,and the cross-modal fusion mechanism is not fully utilized to enhance the feature representation ability.Based on the above problems,the thesis proposes two different improved algorithms to improve the retrieval accuracy of the video moment retrieval task.Firstly,aiming at the problems of roughly extracting modal features in existing methods,the thesis introduces a Fine-grained Cross-modal Fusion Network for video moment retrieval(FCFN).According to the temporal characteristics of video,the enhanced selfattention residual mechanism is used to analyze the fine-grained semantic features of video frames in the stage of video feature extraction,and the correlation between video frames is calculated,so as to obtain richer and more accurate semantic context information of video features.In the text feature extraction stage,the proposed method uses a circular convolutional neural network to encode the query text,and obtains the contextual semantic information before and after each word,thereby preventing the problem of information loss during the encoding process.Then,aiming at the problems of independently processing video and text modalities and the lack of effective multi-modal feature fusion mechanism in the above methods,the thesis proposes a Query Guided Attention Network for video moment retrieval(QGAN).The proposed method extracts video features conditional on the query text,and the specific implementation is to learn the video frame features that capture the closest semantics to the given text description through the scaled dot product attention mechanism.At the same time,the proposed method adopts the multi-modal bilinear fusion mechanism for effective feature fusion to explore the semantic correlation between different modalities.Finally,the two proposed methods are evaluated on two challenging datasets(TACo S and Activity Net-Captions),and compared with the state-of-the-art methods in recent years.The experimental results show that the proposed two methods in thesis are superior to the existing methods,demonstrating their effectiveness in the task of video moment retrieval. |
