Research On Continuous Language Translation Of Sign Video Based On Deep Learning

Sign video translation is a task in the field of computer vision,which targets to translate sign video into natural languages.It can improve the communication quality between deaf and normal people,but also break the semantic gap between different languages.Currently,based on deep learning technology,the development of artificial intelligence facilitates the public life of people.The study of sign video translation is a hot topic under this research background.Based on the classical neural modules,i.e.,Temporal Convolution Network(TCN)and Bidirectional Gated Recurrent Unit(BGRU),this thesis presents a two-stream sequential learning model;it uses TCN to capture short-term correlation information and BGRU to capture long-term context information.After the probability score fusion of these two modules,the model is optimized with the Connectionist Temporal Classification(CTC)function.The Word Error Rate(WER)of the entire model on RWTH-PHOENIX-Weather is 0.3%lower than either stream(sequential learning module).Furthermore,this thesis proposes a pseudo-supervised learning method for sign language translation based on the Expectation Maximization(EM)Algorithm,addressing insufficient supervision(accurate label alignment in sign language videos).In M-step,the model adopts the BGRU with the CTC function to generate pseudo-labels.Then in E-step,the method optimizes the feature extraction model 3D-ResNet with the above-mentioned pseudo-labels.The model is iteratively optimized until it converged.Finally,WER on RWTH-PHOENIX-Weather is 40.9%,which demonstrates the effectiveness of this method.