| Predictive Business Process Monitoring is a technique that builds predictive models based on historical event logs to predict the future behavior of ongoing process instances,thereby optimizing business processes and improving the efficiency of business process management.Predictive Business Process Monitoring includes multiple different prediction tasks,such as next activity prediction and case outcome prediction.In recent years,deep learning methods have received increasing attention in solving these prediction tasks.However,these methods are designed and trained for specific deep learning models and prediction tasks,without considering generality and scalability.To apply them to other prediction tasks,a large amount of labeled data is required as input and the model needs to be rebuilt and retrained.In addition,deep learning methods also have a “black box” nature,that is,their internal structure is complex and difficult to understand,resulting in a lack of transparency and interpretability of their models,making it impossible to explain or verify the mechanism of model operation and prediction results.To solve the next activity prediction task and case outcome prediction task,this paper first defines the next activity prediction model and case outcome prediction model based on the BERT(Bidirectional Encoder Representations from Transformers)model structure.To further improve the generality and scalability of the model,this paper introduces the idea of transfer learning and proposes a novel pre-training task,namely Masked Activity Model,which is transferred to the next activity prediction model and case outcome prediction model for fine-tuning,so as to quickly and effectively apply to multiple different business process scenarios.To enhance the interpretability of the model,this paper takes attention heads as research objects,explores their roles in the model by visualizing or masking them,and analyzes and explains the internal mechanism of the model.The main research contents of this paper include the following aspects:(1)Aiming at the problems of sequence modeling methods based on RNN or LSTM,such as the fixed input time step and the dependence of the current output on the previous output result in low training efficiency and difficulty in capturing long-term dependencies between activities,this paper proposes a BERT-based business process prediction method.By modifying the BERT model structure,the next activity prediction model and case outcome prediction model are defined respectively.These models can use Transformer and attention mechanism to capture global context information and long-term dependencies between activities in traces,and can process all activities in input traces simultaneously without sequential computation to improve training efficiency.Experimental results show that they outperform existing methods on multiple event logs.(2)Aiming at the problem that existing methods are designed for specific models and tasks,lacking generalization ability and unable to be directly applied to other prediction tasks,this paper proposes a multi-task prediction method of business process based on BERT and transfer learning.This method first proposes a new pre-training task for the predictive business process monitoring domain,namely masked activity modeling.This task utilizes the bidirectional Transformer structure in BERT to capture deep bidirectional context information of traces,and finally trains a sufficiently general trace representation.Then,the pre-trained BERT model is transferred to the predefined next activity prediction model and case outcome prediction model respectively,and fine-tuning strategies are used to train these two models separately.Evaluation experiments are conducted on eleven real event logs,and the results show that different masking strategies and masking probabilities in the pre-training task affect the performance of prediction tasks.In addition,the proposed method not only performs well on next activity prediction and case outcome prediction tasks,but also can adapt faster to multiple different prediction tasks with higher prediction accuracy compared with baseline methods.(3)Aiming at the problem that deep learning models are constantly improving in accuracy but have low interpretability,which limits their practical application,this paper proposes an explainable prediction method based on multi-head attention.The method utilizes multi-head attention to “focus” on different behavioral relationships between different activities in the input trajectory from multiple perspectives,and extracts and visualizes the attention head weights directly from the model to observe what the attention heads “focus” on,explore what roles they play in the model,and analyze the internal mechanism of the model by masking attention heads.This paper first verifies the impact of different fine-tuning strategies on prediction performance through eleven event logs,and determines the best fine-tuning strategy for different business process scenarios,and compares with existing methods to prove its effectiveness.Second,this paper masks attention heads by layer and proportion and observes their impact on performance,studies the structure and mechanism of the prediction model,and finds that the prediction model based on multi-head attention mechanism has strong robustness and flexibility.Then,this paper masks attention heads one by one,and visualizes and analyzes their roles and functions in the prediction model.Finally,this paper selects the most representative attention heads,compares their weight distribution changes before and after fine-tuning,and finds that prediction models built with different event logs have different mechanisms and characteristics.Figure [85] Table [26] Reference [87]... |
