Font Size: a A A

Research On Text Classification And SEQ2SEQ Model Based On Involution

Posted on:2022-06-19Degree:MasterType:Thesis
Country:ChinaCandidate:Y S ZhuFull Text:PDF
GTID:2568307022498044Subject:Software engineering
Abstract/Summary:PDF Full Text Request
In natural language processing,the more popular feature extractors are RNN,CNN and Transformer.Among them,since the hidden layer state of RNN at each moment depends on the output at the last moment,RNN is very unfriendly to large-scale parallel computing and inefficient.Both CNN and Transformer are convenient for parallel computation with high efficiency.However,CNN’s ability to acquire semantic and long-distance features is not as good as that of Transformer based on self-attention mechanism,so the performance of THE MODEL based on CNN is usually worse than that of Transformer based on self-attention mechanism in NLP task,especially in seq2 seq sequence generation task.However,the self-attention mechanism is not perfect,because the time complexity based on the sentence length N squared results in the longer the sentence length,the slower the speed.In recent years,many researches are devoted to finding better feature extractors.Recently,a new operator called Involution has been proposed in the field of computer vision.After using it as a feature extractor for images,its effect exceeds that of models based on ordinary convolutional and self-attention mechanisms.In view of its outstanding effect in the field of vision,this paper introduces Involution into the field of natural language processing to replace ordinary convolutional and self-attention mechanisms as text feature extractors.Involution is actually a method of dynamically generating convolution kernel based on time step.Taking it as the basic operator of convolutional neural network is conducive to improving the ability of CNN to acquire semantic information,reaching or even surpasses the level of self-attention mechanism,with smaller parameters and faster speed.Based on this,text classification models Text Inv and DPInv and sequence generation model Inv seq2 seq are proposed in this paper.Text Inv and DPInv both use Involution as text feature extractors.The difference lies in that Text Inv is a shallow model with only one Involution,while DPInv is a deep model,which extracts deeper semantics by stacking multiple layers of Involution and introducing residual join.Inv seq2 seq mainly replaces the self-attention mechanism module of Transformer with Involution and retains the rest of Transformer’s structure.Through experiments,it is proved that Involution in NLP can also achieve better results than ordinary convolution and self-attention mechanisms.In the text classification task,the effect of Text Inv and DPInv exceeds that of Text CNN and DPCNN based on ordinary convolution respectively.In the sequence generation task,the effect of Inv seq2 seq exceeds that of Transformer based on self-attention mechanism.
Keywords/Search Tags:Involution, Self-attention, Convolution Neural Network, Text Classification, Sequence Generation
PDF Full Text Request
Related items