Research On Natural Interaction Models And Algorithms For Fusion Of Virtual And Reality Experiments

Intelligent teaching has always been one of the important research directions of humancomputer interaction,and also a major strategic task of China’s education modernization.The document China’s education modernization 2035 issued by the CPC Central Committee clearly points out that the reform of education should be sped up in the information age.This means that intelligent teaching is in line with the concept of national education development,and is an essential measure to promote China’s education reform.As for the domestic chemical experiment class,the unbalanced distribution of educational resources often leads to the failure of experimental classroom.Schools and teachers take into account the safety of students so that they avoid carrying out dangerous experiments.Teachers often use textbooks and videos to impart knowledge of chemical experiments.These methods cannot endow students with good hands-on ability.The acquirement of chemical experiment knowledge is not deep-seated,merely focusing on book itself.Therefore,under the support of the state key research and development program "Multimodal Natural Interaction of Open Experimental Teaching Environment based on Integration of Virtuality and Reality",this paper explores multimodal interaction mode and experimental interaction mode integrated with virtuality and reality based on the application background of middle school students’ chemical experiment class.The purpose is to establish a natural and harmonious human-computer interaction mode for experiments with the combination of virtuality and reality,improve the ability to understand user’s intention in human-computer interaction,and reduce the cognitive load in user interaction.Aiming at low efficiency of human-computer interaction,low ability of understanding user’s intention and lack of sense of operation and reality of using real experimental equipment in current virtual experimental platform,this paper proposes an intention understanding model based on multimodal interaction,and establishes a virtual reality fusion experimental environment based on multimodal intention understanding.Certain shortcomings of the current virtual experiment platform are solved.The main contents and innovations of this paper are summarized as follows.(1)This paper designs and implements an intelligent experiment kit for middle school chemistry experiment,which provides a new experimental tool for eliminating hazards of experiment in middle school’s chemistry class.In order to meet the basic functions of real experimental equipment,this paper designs and implements a new set of intelligent experimental equipment according to the principle of "combining virtuality and reality,and giving priority to the latter".It proposes an operation interaction algorithm based on intelligent experimental equipment,which is used to sense the operation behavior of tipping and stirring in the experiment.The experimental equipment not only has the basic functions of traditional experimental equipment,but also solves the problem that people are unable to have the sense of operation and reality of using real experimental equipment in most existing virtual experiments.The experimental result shows that the intelligent experimental device can effectively identify user’s operation behavior,enhance his or her sense of reality and operation,and is conducive to improving user experience and teaching effect.(2)Two multi-modal intention understanding models are proposed and implemented,which enhance the ability of intention understanding in virtual and real fusion experiments.In this paper,multimodal fusion perception algorithm(MFPA)and multimodal intention active understanding algorithm(MIAUA)are proposed,which endow the ability of understanding user intention with virtual real fusion experiment.MFPA algorithm combines speech and tactile information in virtual and real fusion experiments.By analyzing user information of different channels and utilizing the characteristics of complementation and mutual exclusion of multichannel information,the multimodal information fusion is realized in the decision-making level,and the real intention of the users is perceived.MIAUA algorithm uses weighted fusion to realize multimodal information fusion in decision-making level,and evaluates the fusion results from two levels.After analyzing and evaluating the result,the human-machine interaction is introduced.The machine actively interacts with the user and requests users to supplement a modal information.The intention understanding ability of virtual reality fusion experiment is improved by computer active interaction,and the interaction among humans is simulated.The user-machine interaction experience is given,and interaction can be further natural and comfortable.The experiment shows that the average accuracy rate of intention understanding for 8 kinds of user intention MFPA algorithm is 95.91%,the mean accuracy rate of intention understanding of MIAUA algorithm is 99.13%,and the MIAUA algorithm is 3.22% higher than MFPA.In general,the multimodal intention understanding model can effectively understand the real intention of users,improve the efficiency of human-computer interaction and reduce user interaction load.(3)The navigation interaction and game interaction mode based on virtual reality fusion experiment are designed and implemented.Teaching navigation is added to the virtual reality fusion experiment to guide users to learn chemical experiments,and to remind and correct wrong operation behaviors in experiments,so as to realize automation when people carry out experiments.In order to solve the problem that most of the existing virtual experiments are boring and short of interest,this paper absorbs the game elements of the game platform,and proposes a game interaction mode according to virtual reality fusion experiment.This interactive mode makes up for the boredom of the traditional experimental mode,stimulates students’ interest and motivation in learning during the experiments,increases their interest in the virtual reality fusion experiment,and improves students’ learning efficiency.