Recommendation Model Based On Self-attention Mechanism And Automatic Mixing Precision Optimization Research

With the development of new technologies such as big data,artificial intelligence and mobile internet,network data has shown an explosive growth trend.The huge amount of data has satisfied users’ demand for information,and users enjoy the convenience that data information brings to their lives.However,the huge amount of information has also created the problem of information overload.Faced with data information of varying quality,it is becoming increasingly difficult for users to obtain information that is useful to them,and the efficiency of information use is declining.Recommendation models have been developed to meet users’ needs for personalised information.Recommendation models can select the most valuable information for users from a large amount of information to improve the efficiency of users’ access to information,among which neural network-based recommendation models have become one of the mainstream recommendation models.The main task of neural network-based recommendation models is to discover user preferences and item features from a large number of items and user reviews,and use these features to recommend items to unknown users.In this thesis,we study the processing of review text by convolutional neural networks and the use of a self-attention factorization machine to process review features and predict ratings.The importance of the user and item feature matrices is discriminated by the self-attention network,and feature weights are automatically assigned by the self-attention mechanism to improve the accuracy of the neural network-based recommendation model.To address the problem that the training time of neural network-based recommendation models is too long and leads to overflow of the device memory,this thesis focuses on optimizing the model training process through automatic hybrid precision training of the model,which works as follows:1.The self-attention network-based recommendation model,SATrans Net,is proposed to address the problem that neural network recommendation models using Factorization Machine cannot discriminate the importance of feature interactions.The SATrans Net model uses convolutional neural networks to process user and item ratings and extract user preference and item feature matrices.The self-attention mechanism is introduced in the feature matrix processing,and the importance of each potential feature is differentiated by automatically assigning different weights to each feature by the self-attention network.The SATrans Net model was experimented with on Amazon’s experimental dataset of eight different categories.The experimental results show that the SATrans Net recommendation model performs better overall with lower Mean Square Error and Mean Absolute Error,higher cumulative gain of normalized discount on different datasets.2.To address the computational and memory requirements of the recommendation model and to reduce the model training time,an Automatic Mixed Precision training recommendation model AMPSA is proposed.The AMPSA model has two layers of logic in the training process,namely Autocast and Grad Scaler.The Autocast logic automatically determines the input data type for each operation based on a given black-and-white list,using a 32-bit floating-point algorithm and a 16-bit floating-point algorithm.The Grad Scaler logic addresses the problem of avoiding floating-point underflow.During model training,the scaling factor is dynamically increased or decreased by checking for overflows in the weight gradient,and the scaling factor is automatically selected to further simplify mixed accuracy training.The AMPSA model has been tested on datasets of varying sizes,ranging from 37 MB to 3 GB.The experimental data show that the proposed model can improve the training speed and reduce the memory consumption by about 26%.