Gesture Design And Recognition Based On Post-WIMP Interfaces

At present, due to the advantages of syntax minimization, object visualization and high real time, the WIMP interface based on desktop metaphor is widely used. However, with the rapid development of computer technology, the disadvantages of WIMP interface, such as the multi-window interface, the operation only focused on by users are increasingly apparent. Accordingly, the next-generation called Post-WIMP has been focused on by the researchers, Due to the Post-WIMP interface of metaphor, the most natural way to interact is selected, and the overall interface design and interaction methods are changed. For these problems, the method of gesture design and recognition based on Post-WIMP interface is proposed. Firstly, the Post-WIMP radiation-tree interface and system based on hierarchical information is designed; Secondly, the gesture is operated with the interface to induce the hierarchy, and the gesture recognition is operated with Kinect and Leap motion device.The main work of this paper is about three aspects:1) The Post-WIMP gesture interface and system for the hierarchical nature of information manipulation is designed. Due to the reason that the current hierarchical interface only attaches importance to the same layer brotherhood relationship and the parent-child relationship between upper and lower layers is ignored. For a more user-friendly operation, the circular tree-graph visualization and 3D view are combined effectively, and the interface for the natural gestures and recognition is constructed.2) The Fingertip detection and gesture recognition of Post-WIMP’s gesture based on Kinect is studied. To achieve the collection of hierarchical gesture data, the depth camera Kinect and the depth image of Kinect to acquire location information of fingertips are used, then the hierarchical gesture is completed with the state machine combined with the palm position and the palm speed.3) The gesture combined with Leap motion based on Post-WIMP interface metaphor is designed and recognized. Due to the metaphorical feature of the Post-WIMP interface, the gesture recognition is not accurate. To increase the tolerance of gestures, a new idea of feature extraction which combines the spontaneous gesture set with the standardized gesture set is proposed, then the test set is used to evaluate the effectiveness of the feature for the training set.