Multi-channel Human-computer Interaction Toolset Design And Development

Human-computer interaction technology is currently widely used in various fields.However,current interaction technologies are diverse and no longer limited to traditional keyboard and mouse.Various natural interaction technologies,such as multi-touch,gestures,eye tracking,body posture,voice,facial expressions,etc,have emerged.This provides flexible means of interaction for various interactive systems,especially virtual reality systems.However,for each application system developed,different devices and program codes are required.Programmers often need to spend time learning how to use and write programs for a new device.Therefore,it is necessary to develop a toolkit that integrates various interaction devices and provides corresponding interaction services to improve development efficiency and save development costs.On the other hand,since many natural interaction technologies belong to "non-preciset"interaction,multi-channel perception fusion is needed to improve interaction efficiency.In this regard,this paper,under the support of the National Key R&D Program "Cloud-based Fusion of Natural Interaction Devices and Tools",conducted relevant research,developed a multi-channel human-computer interaction toolkit,and applied it to multiple VR application systems.The multi-channel human-computer interaction toolkit system supports natural interaction methods such as gesture,eye tracking,touch,body posture,voice,and physiology,and supports various interaction channel fusion modes such as redundancy,complementarity,and mixing.It also provides cross-device consistent interaction processing and presentation interfaces that organize content and adaptively present them based on dynamic changes in environment,devices,and user status.On the one hand,this system,as a medium between virtual reality interaction devices and application scenario developers,can provide users with standardized processed data information from various types of interaction devices,greatly reducing the difficulty and workload of application scenario development design without programming data acquisition and standardization processes.On the other hand,the toolkit system can provide users with multiple interactive virtual reality device data channels at the same time,allowing users to improve the accuracy of human-computer interaction by fusing multiple channel data,helping to enhance users’ immersion,interactivity,and multi-perception in virtual reality application scenarios.This article first analyzes the requirements,outlines the design,and provides detailed design for the multi-channel human-computer interaction toolkit system.In the requirements analysis,the functional and non-functional requirements of the system were analyzed,and an overview of the system was provided.In the outline design,the system architecture was explained.In the detailed design,the logical design of the system and the design of each part of the system’s functions were described in detail.Subsequently,based on the above analysis and design,the system was implemented and tested.The development of this system uses the Unity3D rendering engine,and the programming language used is C#.In terms of human-computer interaction device selection,this system uses various commonly used interaction devices in virtual reality systems,including inertial data gloves and Kinect,as the basic devices for different interaction channels,and designs a universal device module to support users in adding other types of interaction devices.The system realizes the collection and fusion of device data in the virtual reality environment,and transmits the data to the application program developed by the user after the data is standardized and fused.This system has been applied in the actual"Human-Computer Interaction Technology" course teaching.It provides data collection services for interaction devices for student-designed and developed application programs,reduces the difficulty of developing virtual reality application programs for students,and enables students to focus more on logical design and development of interactive systems,saving time for experimental training and improving learning efficiency.In addition,this system has been applied in the development of multiple virtual reality systems:in the mixed reality fish tank interaction application,this system collects the touchpad data,speech recognition data,and body posture device data required by the application program for interactive content design;in the virtual reality roller coaster application,this system collects brainwave device data and physiological signal acquisition device data for analyzing user’s emotional changes.