The Syudy Of Few-shot Slot Filling Algorithms Based On Self-supervised Learning

Slot filling is a task to identify key information from user’s discourse,such as addresses,dates,times,song names,restaurant names,etc.,composing an important module of modern task-oriented dialogue systems.However,due to the time-consuming and laborious data annotation process,it is often difficult for developers to collect enough annotation data for different scenarios.Therefore,the research of few-shot slot filling algorithms has been widely concerned in the industry,i.e.,how to improve the performance of the slot filling models with only limited annotated data.Previously proposed few-shot slot filling algorithms mostly rely on external knowledge,which usually requires expert customization or is difficult to obtain in specific scenarios,limiting the applicability of these algorithms in practical scenarios.Self-supervised learning is a novel learning paradigm popular in recent years,which performs extremely well in large-scale image or text representation learning.It typically relies on the inherent characteristics of unlabeled data to construct the objective function,so that the model can learn directly based on unlabeled data,which solves the problem of lacking labeled data in traditional supervised learning.Inspired by the techniques such as adversarial training,contrastive learning and prompt learning,this paper studies the few-shot slot filling algorithms based on self-supervised learning,attempting to solve the problems encountered in the few-shot slot filling models and improve the model performance by constructing appropriate self-supervised auxiliary tasks.The contributions of this paper are listing as follows.Firstly,this paper enumerates the problems existing in the traditional slot filling algorithms with limited training data,including over-fitting,poor robustness,poor representation learning,and the inconsistency between pre-training and fine-tuning phases.Then,this paper analyzes the bad cases in the experiment and reveals the possible causes of these problems.Secondly,this paper proposes the adversarial semantic decoupling approach to alleviate the difficulty on identifying open-vocabulary slots,easy over-fitting and poor robustness issues.It trains the model to correctly recognize the corresponding slot values on those perturbed samples,thus forcing the model to focus on the global sentence patterns rather than the specific slot values.Experimental results demonstrate that the method avoids over-fitting of local semantics in the few-shot scenarios,improves the generalization and robustness of unknown slot values and improves the final performance.Thirdly,to solve the slot boundary recognition and type recognition errors caused by poor model representation learning,this paper adopts the method of contrastive learning to construct sentence level and slot level contrastive auxiliary objectives respectively.Sentence-level contrastive objective can make the model perceive the whole sentence pattern and reduce boundary recognition errors,while slot-level contrastive objective can improve the slot representation space and alleviate the type confusion caused by semantic similarity of slots.Experiments show that the contrastive learning can significantly improve the representation space,enhancing the cohesion within the same slot type,and reduce the coupling between different slot types.Finally,to solve the inconsistency of the inputs and the training objectives between pre-training and fine-tuning phases,this paper studies the self-supervised learning method of prompt learning.By converting the slot filling task as a sequence-to-sequence generation task,fine-tuning with generative language models and introducing learnable soft prompts,the method can guide the model to fully express the semantic knowledge learned in the pre-training stage during fine-tuning.Experiments show that this method makes the expression of slot filling task more natural,and thus improve the performance of few-shot slot filling task.