Research On Key Technologies Of Sign Language And Speech Translation Service Robo

With the successful application of deep learning technology in the field of sign language recognition,it is possible to alleviate the communication barrier that has been perplexing the hearing-impaired people.In order to apply sign language recognition technology to daily life and effectively alleviate the communication difficulties between hearing-impaired people and normal people,this paper makes research on sign language and speech translation system.In order to realize the function of sign language speech translation,this paper first studies the algorithms of isolated word recognition and continuous sign language recognition.For sign language isolated word recognition,a sign language isolated word recognition algorithm based on C3 d is proposed.For continuous sign language recognition,we create a continuous sign language recognition algorithm based on global attention mechanism and LSTM.Then,based on i FLYTEK’s open platform,the voice and text mapping is established to realize the task of voice and text translation.Finally,the above research contents are carried on the service robot platform to build the sign language and voice translation system,and realize the sign language and voice translation function in a limited range.1)An isolated word recognition algorithm of sign language based on C3 d is proposed.K-means clustering is used to extract the key frames of sign language video,remove the redundant parts of the video,segment the extracted key frame pictures,highlight the hand area,and facilitate the C3 d network to extract its motion trajectory.The two kinds of data are spliced into new videos and input into the network for training.The hand shape features and hand trajectory features extracted from the two forms are fused and classified by SVM.The algorithm is tested on 80 CSL sign language isolated word data sets,and the average recognition accuracy is 76.625%.2)Continuous sign language recognition based on global attention mechanism and LSTM is proposed.The main body of the network structure adopts the codec structure.In the data preprocessing stage,the difference method is used to extract the video key frames,remove the redundant frames and reduce the influence of noise in the recognition process.RESNET is used as the encoder for feature extraction,which can well preserve the timing of data when effective features are extracted.Using the LSTM network integrating the global attention mechanism as the decoder,the timing analysis is carried out through the LSTM,and the recognition results are continuously optimized in combination with the historical context information.The average recognition rate of the algorithm on the Chinese continuous sign language data set CSL is 90.08%,and the average word error rate is 41.2%.Compared with five sign language recognition algorithms,this method has advantages in recognition accuracy and translation performance.3)Service robot prototype system integration of sign language and speech translation.By carrying the above algorithm and the designed speech recognition module on the service robot,a system that can realize the task of limited sign language and speech translation is constructed.