On Knowledge Driven Natural Language Reasoning Models

In recent years,deep learning models have demonstrated remarkable performance in downstream tasks of natural language processing.Particularly,with the advancement of pre-trained language models,such models have dominated the leaderboards in various evaluation tasks.However,existing pre-trained models still exhibit several significant limitations: Firstly,pre-trained models perform poorly when it comes to integrating knowledge for reasoning,leading to limited performance in handling complex problems and difficulty in generalizing to new issues,thus reducing their reliability in practical applications.Secondly,deep learning models require large amounts of labeled data,which restricts their generalizability in scenarios with scarce labeled data.For many niche domains and low-resource language tasks,obtaining sufficient high-quality labeled data is challenging.Lastly,deep learning models have weak interpretability,making it difficult for researchers and users to understand their internal mechanism,which in turn affects the models’ performance in terms of safety,reliability,and fairness.To address these issues,this study focuses on knowledge-driven natural language reasoning models.By combining pre-trained models with knowledge reasoning,we aim to improve the models’ reasoning performance and interpretability.Simultaneously,the learning and transfer generalization of knowledge can also help to enhance the models’ generalizability and alleviate data scarcity issues.Therefore,the research on knowledgedriven natural language reasoning models is of great significance.Therefore,this paper focuses on an important scientific question: How to combine symbolic knowledge with pre-trained models to construct reasoning models? To address this issue,our research conducts an in-depth investigation into challenges at both the model and data levels: Model-level challenges mainly manifest as difficulties for neural networks to model symbolic complex reasoning processes.The knowledge learned by neural networks during training is implicitly embedded in the weights,lacking a clear structure.This restricts the ability of neural networks to represent discrete symbols and logical structures,as well as complicating the execution of rule-based complex reasoning tasks.Data-level challenges lie in the difficulty of obtaining labeled data for reasoning tasks,leading to poor generalization performance of the models.In many cases,complex reasoning tasks require a large amount of labeled data to train neural networks.However,such data can be challenging to acquire,especially for tasks involving domain-specific expertise.To overcome the model-level challenges,we propose three research studies in this paper.These studies,by constructing neural-symbolic reasoning models that combine symbolic knowledge with neural networks,effectively improve the models’ performance in complex reasoning tasks:(1)Reasoning model based on unstructured knowledge(text):This study uses the fact verification task as the scenario to investigate how to retrieve and represent textual knowledge and construct a text-based reasoning model.Firstly,in terms of knowledge representation,we propose using semi-structured knowledge semantic graphs to represent factual structures in unstructured textual knowledge.Secondly,based on the knowledge semantic graph,this study combines pre-trained models with graph neural networks to construct a neural-symbolic reasoning model that jointly learns factual structures.Experimental results show that our approach can improve the performance of the model and increase the interpretability of the predictions.(2)Reasoning model based on structured knowledge: This study focuses on the table-based fact verification task,aiming to explore multi-modal graph-enhanced knowledge representation and a neural-symbolic reasoning model based on neural modular networks.By modeling table structure,cross-modal knowledge semantic associations,and compositional semantics of logical functions,the model’s performance and interpretability are improved.(3)Reasoning model based on combined knowledge: This study uses table-text open-domain question answering as an application scenario and proposes a neural-symbolic reasoning model based on combined knowledge.To address cross-modal knowledge reasoning issues,this study introduces an evidence-chain-based neural-symbolic reasoning model.This model extracts symbolic reasoning paths from combined knowledge as reasoning prompts and combines them with pre-trained models for making predictions,thereby enhancing the model’s performance and interpretability.To alleviate data sparsity issues and further improve reasoning performance,this study also proposes a generative reasoning pre-training task.In addition,to address the data-level challenges,this paper proposes the following two research works.Experiments demonstrate that these methods have superior performance and generalization capabilities in multiple reasoning tasks:(4)Self-supervised reasoning pre-training method based on logical knowledge: Logical knowledge represents logical reasoning rules and causal relationship inference abilities.Utilizing logical knowledge helps models solve complex reasoning tasks.This study establishes a logical knowledge-driven pre-training method: firstly,by automatically constructing a large-scale self-supervised pre-training corpus using logical indicator words,enabling the model to learn logical knowledge and alleviate the data sparsity issue in logical reasoning.Secondly,a more computationally efficient adversarial logic pre-training method is proposed,further enhancing the model’s logical reasoning capabilities.Experiments show that the model has superior performance and generalization abilities in 16 reasoning tasks.(5)Multi-task reasoning pre-training method based on task knowledge generalization: This study aims to construct a multi-task general model so that knowledge from different tasks can be shared,improving the model’s generalization performance on low-resource tasks.This research proposes a learnable and flexibly dynamic combination of template-based prompting and multi-task pre-training,transforming task input and output into a unified form,supporting multi-task learning and generalization transfer.Experiments on 30 natural language tasks demonstrate that,by automatically constructing learnable task templates and multi-task pre-training,shared knowledge and task characteristics can be better modeled,reducing the amount of data required for model generalization in downstream tasks.In summary,based on various types of knowledge,such as unstructured knowledge,structured knowledge,combined knowledge,logical knowledge,and task knowledge,this paper explores how to address challenges in constructing knowledge reasoning models,including challenges at the model and data levels.The challenges at the model level involve using neural networks to model symbolic reasoning processes,while those at the data level pertain to the scarcity of labeled data for reasoning tasks.Experiments show that knowledge reasoning is of great importance for improving model performance,interpretability,and generalizability.The main contributions of this paper include:(1)constructing neural-symbolic reasoning models,enhancing the model’s complex reasoning performance and prediction interpretability;(2)using knowledge-driven reasoning pretraining to alleviate the labeled data scarcity issue and improve generalization performance in downstream tasks.