Research On Automatic User Interface Code Generation Based On Deep Learning

In the front-end development of the software development process,front-end developers need to write the corresponding interface code based on the UI(User Interface)design images.In order to restore the position and style of UI components in the UI design perfectly,developers usually use a trial-and-error approach to write the interface code,which is a time-consuming and repetitive task.Implementing automatic transformation from UI design images to interface code can avoid this repetitive trial-and-error process,improve development efficiency,and facilitate rapid software iteration.In recent years,a number of works have been dedicated to automatically transforming UI design images into the corresponding interface code using deep learning techniques.In these studies,the deep learning models proposed by researchers are basically common structures in the image caption domain,which combine CNN(Convolutional Neural Network)and RNN(Recurrent Neural Network)to generate the corresponding interface codes based on the input UI design images.However,the generation of interface code is different from the generation of natural language,because interface code is long text with nested relationships.RNN-based models not only cannot extract the hierarchical structure information in the code,but also have limited ability to model the long-term dependencies of long text because of the problems of gradient explosion and gradient vanish.Finally,in the field of web interface code auto-generation,the existing public datasets have the defects of too simple samples and too few samples,and the generated interfaces have a single style and fixed structure.To solve the above problems,the main work and innovations of this paper are as follows:(1)In order to enable the deep learning model to generate richer and more diverse interface codes,this paper creates a new dataset that supports the generation of UI components in more kinds,more numbers,and more combinatorial ways.And a subsequent study is conducted on the existing dataset and the dataset proposed in this paper to demonstrate the strong generalization of the deep learning model proposed in this paper.(2)In order to avoid the problems of gradient disappearance and low training efficiency brought by recurrent neural networks,an interface code generation model based on temporal convolutional networks is proposed in this paper.In existing studies,recurrent neural networks are usually used to model user interface codes,considering the special characteristics of user interface codes,whose long length and strict hierarchical structure are not suitable for modeling using recurrent neural network.In this paper,we propose a temporal convolutional networkbased interface code generation model for the task of modeling user interface code,which is able to model hierarchical relationships and long-term dependencies in the code using onedimensional convolutional layers and causal convolution approach.In addition,this paper combines a hierarchical attention mechanism to emphasize the textual features at different levels in the interface code.After experiments on existing datasets and the dataset created in this paper,and comparing with the current state of the art models in the field of interface code generation,the model proposed in this paper performs better in several metrics such as BLEU and error rate,and the model proposed in this paper only accounts for 14.5% of the existing models in terms of parameters,which is a lighter and more efficient model.(3)In order to solve the inconsistency problem of the model during training and inference,i.e.,Exposure Bias,a schedule sampling training method applied to temporal convolutional networks is proposed in this paper,and negative samples are created through a twice-decoded approach.The model is fully aware of the negative samples during training,which makes it possible to correctly handle the negative samples in the inference behavior of autoregression and eliminate the influence of Exposure Bias to a certain extent.After experiments,it is found that the BLEU scores of the model using the schedule sampling training method are improved by about 4% on the test set,especially in the case that the target code sequence length is greater than 100,the BLEU scores of the codes generated by the schedule sampling method are all higher than those of the ordinary Teacher-Forcing training method.(4)To solve the problem of repetitive code segment generation when using greedy search or beam search for code generation,this paper proposes a random strategy-based approach for interface code generation and compares three different random strategies with the traditional method of maximizing conditional probability.On a test set of size 250,the proposed method reduces the cases of repetitive code segment generation by about 40% and effectively reduces the error rate of generated code while improving the BLEU score by about 2%.