Research On Multilingual Simultaneous End-to-End Speech Translation

Speech translation is a technology that uses computer to translate speech in one language into text in another language or speech,it has many application scenarios.End-to-end speech translation directly translates the speech of the source language into the text of the target language through one model.Compared with the traditional cascading speech translation system,end-to-end speech translation model has the potential advantages of lower latency,smaller model size and less error propagation.In recent years,many representative works have been made on end-to-end speech translation.We analyzes and release the cross-modal and cross-lingual challenges in end-to-end offline speech translation,and explores the multilingual simultaneous endto-end speech translation.Firstly,in offline end-to-end speech translation,a systematic analysis shows that there are big differences between cross-modal and cross-lingual challenges.In view of the above problems,we proposes a model to reduce cross-modal and cross-lingual barriers in end-to-end speech translation,and explores how to dynamically calculate the state of the output at the decoder to obtain better semantic information and improve the translation quality.Secondly,for the application scenarios of real-time speech translation,we research end-to-end speech translation in multi-lingual real-time scenarios,explores the feasibility of real-time translation into different target languages using one model,and designs and implements simultaneous speech translation models based on multilingual and multi-decoder and multi-language single decoder.In this paper,we also design a multi-lingual real-time dateset and a test tool for multi-lingual simultaneous decoding.