A Study On Sequence-to-Sequence Modeling For Symbolic Information Processing

In the past two decades,benefiting from the improvement of computer science and the rise of deep learning,the research of Symbolic Information Processing,especially,Natural Language Processing(NLP),focuses on the symbolic text itself to learn the latent rules and features instead of relying on manual rules provided by experts.Since most NLP tasks address symbolic text sequences,the sequence-to-sequence(Seq2Seq)model structure has become the most common model in Symbolic Information Processing.However,although simple and general,the existing forms of sequence-tosequence modeling have not been adapted and improved for specific tasks.Therefore,this dissertation will systematically study sequence-tosequence models in the field of symbolic information processing,focusing on three tasks: neural machine translation,dependency parsing,and organic chemical prediction.The main work and innovations include the following three aspects:First,this dissertation proposes a Seq2 Seq model based on the extended information to improve the standard Neural Machine Translation(NMT)model,which translates the sentence independently without considering any context information.We observed that humans would search for relevant contextual information in the ”memory” during the translation process and proposed a context-aware NMT model based on Memory networks.We also proposed a document-level NMT model based on global and local document embedding to take the contextual information near the current sentence along with the topical content of the document into account during the translation process.In these ways,the model can perceive contextual information and enhance performance.Second,this dissertation researches the application of the Seq2 Seq model in other tasks in NLP,such as Dependency Syntactic Parsing.Addressing the non-linear output structure,we convert the syntactic tree into the syntactic tag sequence.Thus,it can take advantage of the Seq2 Seq model and be combined with the tri-training method to realize the domain adaptation.Finally,this dissertation extends the application scope of the Seq2 Seq model to “generalized symbolic system”,no longer limited to NLP.SMILES provides a linear notation method for organic chemical reactions.By this means,chemical reactions can be regarded as a special ”language” and can also be processed by the methods commonly used in NLP.Addressing three tasks in organic chemistry(Practicality Judgment,Yield Prediction,and Reaction Prediction),we used edit distance,segmentation,and word embedding training to learn the chemical reaction representation.We also applied the most popular Seq2 Seq model—Transformer model into the reaction prediction task.According to the characteristics of long sequences in chemical reaction texts,we proposed the Multi-ranged Transformer Model to tackle the lack of long-range dependency.