Attention-based Short Text Classification

Short text classification is a task of classifying the polarity of short texts based on their semantic representation,and its application scenarios include people's real-time reviews,discussions,evaluation of products/works on social platforms.These short texts are usually oral and brief.Currently,deep neural networks have become an effective method for short text classification.Specifically,the method based on deep neural network aims to learn a text semantic representation,which directly determines the effectiveness of the deep learning method.However,due to the problem of semantic sparseness in short texts,learning a high-quality representation covering the complete semantics is extremely difficult.To further address the aforementioned limitation,this thesis proposes attention-based joint representation learning network(AJRLN),which proposes the multi-module semantic extraction mechanism based on the characteristics of short text scenes,and introduces a regularization term which equip the model with the ability to extract different attentive features of the sentence embedding.To further generate complete semantic representation,AJRLN utilizes an attention mechanism on the differentiated attentive features for automatically obtaining the joint representation of current sentence.However,AJRLN is a model based on the traditional word embedding and neural network,while the former will cause the model to be restricted by “polysemy” and the latter may cause "structure dependence" problem.In addition,this thesis further proposes a BERT-based sentence representation learning network(BSRLN).It consists of a pre-trained BERT and multi-head self-attention mechanism,which replaces the traditional neural network structure used in AJRLN.Thus,BSRLN further extracts the multi-way correlations within the semantic information for fusion,thereby improves the performance of classification.A series of experiments on real short text datasets demonstrate the validity of AJRLN and BSRLN.The experimental results show that proposed models are obviously better than the original model,and attention-based semantic extraction framework can actually enrich the semantics in short text classification,which exhibits the high availability of these models.