Research On Theory And Technology Of User Experience Improvement For Multi-Modal Services

With the rapid development of multimedia communication technology,multi-modal services of“hearing,seeing,and touching”,integrating emerging haptic signals into traditional audio-visual services,will become the mainstream of multimedia services in 6G era.Currently,the booming multi-modal services emphasize not only the network Quality of Service(Qo S)such as bandwidth and delay,but also user experience.Therefore,how to characterize and improve user experience for multi-modal services are of significant importance.However,due to the strong subjectivity of user experience,as well as stringent and diverse technical requirements of multi-modal services,there are still many technical challenges in existing works on user experience evaluation,user experience improvement,and user experience-driven system optimization.To this end,this paper conducts the research and discussion on the theory and technology of user experience improvement for various types of multi-modal services.Specifically,the contributions of this paper can be summarized as the following four aspects:Firstly,in order to resolve the difficulty of objectively characterizing user subjective experience,this paper proposes a personalized Quality of Experience(Qo E)improvement strategy for audio/video-dominated multi-modal services.On the one hand,by making full use of many time-varying influencing factors on Qo E,from three dimensions of user,context,and device,a deep learning-based personalized characteristic extraction scheme is designed to objectively and precisely characterize personalized Qo E.On the other hand,considering the actual condition of data sparsity for a single user,this paper puts forward a personalized Qo E improvement method based on federated learning,which avoids the risk of privacy leakage through exchanging model parameters among users.Then,in order to resolve the difficulty that traditional Qo E can hardly vividly characterize user’s engagement and presence,this paper proposes a novel evaluation paradigm of user experience(i.e.,immersive experience)for interactive network services of multi-modal services,and then studies the lightweight evaluation methods for immersive experience.On the one hand,by analyzing the differences between traditional Qo E and immersive experience from both the theoretical and technical perspectives,this paper clarifies the necessity and corresponding technical challenges of introducing this novel evaluation paradigm.On the other hand,in order to meet the stringent delay constraints of interactive network services,by exploring the mathematical relationship between influencing factors and immersive experience,this paper proposes lightweight evaluation methods based on artificial intelligence for two scenarios with sufficient data and sparse data,respectively.Subsequently,in order to resolve the difficulty of dynamic improvement of user’s instant experience,this paper proposes an online improvement strategy of immersive experience for interactive network services.On the one hand,under the guidance of multi-domain collaboration,this paper significantly and efficiently improves immersive experience in the context of high-dimensional data by combing network resource scheduling with user profile,device specification,and application type.On the other hand,this paper derives the theoretical bound of immersive experience improvement,which can guide the decision-maker to improve IE to the maximum extent in a more efficient manner.Finally,in order to resolve the difficulty of simultaneously satisfying diverse transmission requirements of multi-modal services,taking remote healthcare as the specific application scenario,this paper proposes a stream transmission strategy based on cross-modal communications.On the one hand,with the aid of common semantics from audio/video signals,a cross-modal haptic reconstruction framework is proposed to improve haptic fidelity at the receiver,which can compensate for its reliability loss during the transmission.On the other hand,by taking full advantage of edge computing and network slicing technologies,this paper designs a heterogeneous stream joint scheduling strategy based on the estimated arrival time of haptic streams to improve the throughput of audio/video streams.Moreover,a remote throat sampling platform is constructed to validate that the proposed strategy can simultaneously satisfy the transmission requirements of low latency,high reliability and high throughput.