Research On Synchronization Algorithm Of Video And Haptic Signals In Multimedia Communication

With the rapid development of science and technology,people’s perception of multimedia is increasing.Haptic media is a new type of media that can greatly enrich the user’s immersive perception experience.Therefore,multimedia systems that integrate haptic media have gradually become a hot spot.This type of system involves the transmission and display of multiple medias in practice.In this process,network and other factors may cause unsynchronization bewteen media streams,which affects the actual quality of user experience.The signal synchronization algorithm is an effective way to solve the above problems,but the current research in this field mainly focuses on the synchronization between audio and video signals,and there is very little research on the synchronization between video and haptic signals(referred to as video-haptic signals),so it is necessary to follow the interactive correlation of the video-haptic signals to research synchronization algorithm of video-haptic signals,so that to ensure the synchronization performance of multimedia communication.Based on the above background,this thesis uses deep learning and computer vision technology to design and implement a video-haptic signals synchronization algorithm that does not rely on timestamps.The specific research content is as follows:(1)Based on the analysis of the correlation characteristics of the video-haptic signals,a synchronization algorithm of the video-haptic signals that does not depend on the timestamp is proposed.The video-haptic signals synchronization algorithm includes signal synchronization detection and synchronization adjustment.The former is used to detect whether the video-haptic signals are synchronized,and the latter is used to adjust the signals to ensure signal synchronization.This thesis first analyzes the correlation characteristics of the video-haptic signals,and then uses deep learning and computer vision methods to extract the feedback force characteristics in the video signal,so as to obtain the characteristic appearance time,and then jointly compare with the feedback force characteristic appearance time of the haptic signal to complete synchronization detection.Finally,using the signal synchronization threshold,an adjustment algorithm based on the synchronization threshold is proposed,which restores the video-haptic signals synchronization by adjusting the video signal when the delay exceeds the threshold(video-haptic signals are out-of-step).(2)Use the forced choice method to design a subjective experiment to analyze the impact of different delays on the user experience,and determine the synchronization threshold on this basis to ensure the performance of the algorithm.The synchronization threshold is closely related to signal synchronization.If the delay between signals is within the threshold,it is considered that the video-haptic signals are in a synchronized state,and vice versa,it is considered that the videohaptic signals are out of synchronization.Obviously,the performance of synchronization algorithm largely depends on the accuracy of threshold.In order to determine the threshold more accurately,while considering the subjectivity of the user experience of video-haptic communication,this thesis uses the forced choice method to design subjective experiments based on a self-built video-haptic signals transmission simulation platform,and determines the synchronization threshold according to the experimental results.The threshold obtained by this method is more in line with the user’s actual subjective feelings,thereby effectively improving the performance of the video-haptic signals synchronization algorithm.(3)Implement the proposed synchronization algorithm and perform performance analysis.In order to verify the effectiveness of the synchronization algorithm,firstly,with the help of the video-haptic signals transmission simulation platform built in(2)and the obtained synchronization threshold,the video-haptic signals synchronization algorithm proposed in(1)is completed.Secondly,design experiments verify the effectiveness of the synchronization detection algorithm for video-haptic signals,and the results show that the synchronization detection algorithm has good detection accuracy.Finally,objective and subjective experiments are desighed in this thesis to verify the overall performance of the proposed synchronization algorithm.The experimental results show that the synchronization algorithm can effectively ensure signal synchronization and improve the quality of user experience.The video-haptic signals synchronization algorithm proposed in this thesis can ensure that the video-haptic signals are displayed synchronously at the signal receiving end while ensuring the quality of user experience,which has certain practical value.