Research On Gesture-Based Development Environment For Natural User Interface And Its Implementation

With the development of mobile interactive technology, the traditional human-computer interaction way is being replaced by more and more natural new interactive ways. In the mobile field, with the disappearance of physical buttons, gesture interaction has become the mainstream of the information input mode, at present, mobile gesture interaction is no longer just a simple touch gestures, Gesture interaction has been developed towards the direction of diversification, User can perform more natural human-computer dialogue with interactive systems through different types of gestures interaction.In the practical process of building gesture user interfaces, we often need to mix a variety of gestures interactive technology to building a gesture user interfaces, because of the diversity of gesture interaction technology, user interface which is integration of a variety of gesture recognition technology development and maintenance process becomes very complicated and difficult, developers will encounter low gesture user interface development efficiency, low maintainability and many other issues. The reason is the lack of efficient and easy to use gesture user interface support tools, this is an urgent problem that developers are facing.First of all, this paper analyzes the characteristics of gesture interaction in the mobile scenario, analysis and research of the theory of gesture natural user interface. Then we collected some gesture that often used in mobile applications, at the same time, we make full use of the various sensors provided by mobile device, designed and implemented some new gestures, we constructed a multi-class set of gestures. On the basis of the gesture set, we propose a gesture user interface description language based on XML, language can be unified description for different categories of gestures in user interface, it can describe the binding relationship which is between interface elements and gestures, at the same time, it can be affirmed that gesture operation corresponding semantic feedback. Design and implement a gestural interface development environment based on the Android platform, this language can support the natural user interface generic metaphor OCGM (Object, Containers, Gesture, Manipulation) design. This language can be unified description for different categories of gestures in user interface, it can describe the binding relationship which is between interface elements and gestures, at the same time, it can be affirmed that gesture operation corresponding semantic feedback. We design and implement a gestural interface development environment based on the Android platform, It mainly consists of three parts, visual interface design and generation tools, gesture library and run-time mechanism, this development environment will help designers and developers to quickly build a gesture natural user interface with multi-response characteristics, it can improve the development efficiency of mobile application, and shorten updated iteration cycle of mobile applications.