Research Of Chinese Sign Language Recognition And Exploration Of Rehabilitation Application Based On Surface Electromyogram

Hand gesture, as the most commonly used body gesture, is widely used in all aspects of human life due to its rich and varied meanings and flexible and convenient way of execution. Hand gesture recognition is known as the process that computer automatically capture, analyze and understand various types of gestures to determine human intentions and to provide the corresponding services. As the advance of modern technology and the improvement of human living standard, gesture recognition becomes a research focus in the fields of human computer interaction, sign language recognition (SLR), rehabilitation training and sports medicine etc..Arbitrary hand movements are completed by groups of muscles which are coordinated and work closely together under the control of the nervous system. The surface electromyographic (SEMG) signal is regarded as one kind of important bioelectric signals caused by muscular activities. SEMG sensor can capture the information of muscular activities, which not only reflect the state and strength of flexion and extension of the joints, but also reflect the information of limb postures and positions. The SEMG processing technologies provide us with important opportunities to capture hand gestures. At the same time, the accelerometer sensor placed on the limb can record the acceleration (ACC) signal, which reflects the position and the trajectory of the hand. The ACC processing technologies provide us another opportunity to capture hand gestures.This work investigates the detection and recognition of various kinds of hand gestures based on SEMG signals. On the one hand, a combined SEMG and ACC approach for large vocabulary continuous Chinese SLR is proposed, which provides a practical solution for the deaf and the health communication and improves the deaf living standard. On the other hand, the gesture recognition technology based on SEMG can be extended to the field of rehabilitation engineering. The recognition result of hand gesture can aid individuals with neuromuscular diseases in rehabilitation training. The main work and achievements of the dissertation focuses on the following aspects:1. Chinese sign language (CSL) alphabet gesture recognition based on the multi-channel SEMG. This study aims at realizing a SEMG-based hand gesture recognition method and exploring a set of gesture normalization schemes, which could improve the gesture recognition result. The main work are as follows:1) A SEMG-based hand gesture recognition framework including signal measurement, active segmentation, feature extraction and classification were proposed and used to classify30kinds of CSL alphabet gestures.2) According to the analysis of sign language gesture process and activities of relevant muscles, a set of gesture definition improvement and action normalization schemes were proposed.3) The results of user testing experiments showed that the average recognition accuracies of30CSL alphabets were improved after user learned the action normalization schemes.4) A novel signal filtering method combined with independent component analysis and adaptive filtering was proposed. The experiments demonstrated that this method can remove the noise of electrocardiography (ECG) from SEMG signals.2. CSL isolated word recognition based on the information fusion of SEMG and ACC. This study aiming at investigating the hand gesture recognition technique based on the SEMG and ACC signals. Considering the complementary characteristics of the two kinds of signals, a hand gesture recognition framework with joint multi-stream hidden markov model (HMM) and decision tree based on the information fusion of SEMG and ACC was presented. Classification tasks were conducted on30CSL one-hand isolated words and121CSL two-hand isolated words, and the experimental results demonstrated that the proposed method can shorten the train and recognition time and improve the recognition accuracy.3. Vocabulary scalable and continuous CSL recognition based on the SEMG and ACC information. The purpose of this work is to propose a vocabulary scalable CSL recognition method and to realize the recognition of continuous CSL, based on the SEMG and ACC data, three basic components can be extracted from each hand gesture. The three components are the hand shape, orientation and trajectory. The CSL isolated word can be recognized by fusing the components recognition results and the CSL sentence can be recognized by assemble the CSL isolated words. Experimental results on the recognition of200CSL sentences composed by120frequently used CSL isolated words demonstrated the CSL gestures could be well recognized by this component-based method. Statistical language model and syntactic model were used to detect and correct continuous CSL error in this study and got a good result.4. Exploration on the SEMG-based gesture recognition application in rehabilitation engineering. Gesture recognition results were used to aid individuals with neuromuscular diseases in rehabilitation training. A novel framework including feature extraction, feature dimensionality reduction and classification were proposed and used to classify20different arm, hand, and finger movements performed by stroke survivors. High gesture recognition accuracies were obtained using high-density SEMG sensors. A series of practical issues were investigated in this study for practical application. The issues include SEMG electrode configuration, channel reduction, and the appropriate choice of some SEMG signal conditioning and preprocessing parameters such as window length, sampling rate and high-pass cut-off frequency. The outcomes reported in this study can be regarded as guidelines for developing myoelectric control systems toward stroke rehabilitation.The research is supported by the National High Technology Research and Development Program of China (The863Program)"Research on the Gesture Input Devices Based on the Accelerometers and Surface EMG sensors"(2009AA01Z322), National Natural Science Foundation of China "Chinese Sign Language Recognition based on Surface Electromyogram"(60703069), cooperation projects with NOKIA Research Center (Helsinki&Beijing) and Graduate Innovation Foundation of USTC.