Abstractive Document Summarization Based On Deep Sequence To Sequence Model

With the rapid development of information technology,explosive growth of information puts higher demands on people’s technology for processing information.Now technology cannot effectively meet the timeliness of information utilization.As a technology to reduce information overload and improve information utilization efficiency,automatic text abstraction has a wide range of applications in practice,such as automatically generating news articles and technical article titles,abstracts,automatic generation of search engine search results,and writing robots.Wait.Prior to the 1990 s,automatic text summaries were dominated by unsupervised techniques.Due to the lack of effective monitoring mechanisms and limited computing resources at the time,these methods were generally less effective in the 1990 s,with machine learning techniques.The emergence of the model learning process through tag data,supervised methods usually have better performance than unsupervised technology,but this method requires a large number of artificial participation in the construction of features.Recently,with the dramatic improvement of computer performance,deep learning technology has achieved very good results in many tasks,especially in unstructured data processing,such as natural language,image,audio and video data types.Compared with traditional machine learning technology,deep learning technology has two advantages:First,deep learning technology can automatically extract features,reduce the researcher’s work of building features,and enable researchers to build system applications faster;second,depth Learning techniques are good at learning low-dimensional feature representations of data,such as word vectors Glove,Word2 vec,etc.,which to some extent make up for the shortcomings of traditional machine learning algorithms.From the generation mode,the automatic text summary task can be divided into two categories,extraction and generation.The abstract is to extract some sentences to construct the abstract by measuring the importance of the sentences in the original text,while the generated abstract is to generate a summary based on the original content by semantic understanding of the original text.Compared to the extraction,the generator is undoubtedly more in line with and close to the human abstract process,but the generative abstraction technique is more complicated and more difficult.In this paper,a series of research work is carried out on the problem of generative abstraction,and how to use the deep learning model to improve the performance of automatic summary of generated text.In addition,functionally,keywords(phrases)are similar to text summaries,which are the general expression of the original content.Therefore,this paper regards keywords as a special form of text summary.The research content of this paper mainly includes two aspects: one is the research on automatic generation of generative keywords,the other is the research on automatic generation of generative text summaries.The main work and innovation of this paper can be summarized as follows:(1)In order to improve the efficiency of automatic keyword generation,a neural network model based on convolutional neural network is proposed to model the context of text sequence.For the task of generating keywords automatically,some researchers have proposed to use the cyclic neural network model to model the context relationship of text sequence.Although this method can express the semantic understanding of the original content better,it is relatively inefficient.Because the calculation of the cyclic neural network in each time step depends on the result of the previous time step,the computer can not process it in parallel.Convolutional neural network is used to replace cyclic neural network in text.Although convolutional neural network is seldom used in sequence model,it can generate vector representation for fixed-length context.By continuously adding convolution layer,it can also play a role in modeling the context relationship of text sequence.Specifically,this paper uses a fully convoluted Sequence-to-Sequence model structure,that is,the encoder and decoder are both convolutional neural networks.In addition,attention mechanism is added to the model structure to strengthen the relationship between the original text and the abstract.A replication mechanism is introduced to generate words that are not in the vocabulary,namely OOV(Out of Vocabulary).(2)Aiming at the task of generating keywords automatically,this paper proposes a neural network model which uses coverage mechanism to model the relationship between generated keywords,so as to improve the accuracy of automatic keyword generation.In the last part,we use the full convolution Sequence-to-Sequence structure,focusing on improving the efficiency of automatic keyword generation.In order to improve the accuracy of the model,this part regards the cyclic neural network as the coding basis of the Sequence-to-Sequence model,and also adopts the replication mechanism to deal with the OOV problem.In addition,the model introduces a replication mechanism to model the relationship between the generated keywords,which can make the generated keywords not only reflect the key information in the original text more comprehensively,but also make the keywords have less information redundancy.By using cyclic neural network,replication mechanism and coverage mechanism,the model has achieved better performance in a number of experiments.(3)In order to improve the accuracy of the task of automatic summarization of medium and long text,a neural network model based on hierarchical attention mechanism is proposed in this paper.In fact,Generative Text Summarization technology based on Sequence-to-Sequence is generally only suitable for short text processing,but has limited processing power for long input text.The reason is that the long input sequence will lead to the long-distance dependence problem,and the encoder can not accurately express the semantic information of the input text.In practical natural language processing applications,the significance of summarizing short text is very limited.Users are more concerned about the ability of long text summarization to help them improve the efficiency of reading and acquiring information.Attention mechanism is one of the key technologies to solve the problem that long text cannot be effectively coded.In this paper,we use location information to improve the existing attention mechanism,and propose a text summary model based on hierarchical attention mechanism,which can differentiate text in different positions.(4)In order to improve the accuracy of extractive summarization task,a neural network model based on generative framework is proposed for extractive summarization task.Although in recent years,with the development of in-depth learning,semantic understanding technology has been rapidly improved,but it is undeniable that automatic abstraction technology for medium and long text still has a long way to go.Comparing with extractive technology,the generative summary model has many problems,such as high computational resource consumption,high time overhead and unsatisfactory accuracy.Most of the traditional extraction methods use optimization methods to search for key sentences in text,and construct abstracts by ranking the importance of sentences.This method does not effectively use the semantic information of the original text.This paper attempts to use Sequence-to-Sequence generative framework to model the semantic relationship between the original text and the abstract,and then uses classification model to extract the sentences in the original text.