Automatic Generation Of GUI Code Based On Deep Learning

Developing GUI-based applications is a quick,iterative process of prototyping,coding and user testing,implementing automatic code generation model will help advance this process obviously.Generating source code from design drawings is a new research field.The existing pix2 code only generates code based on screenshots,feeds the model with same image at each time step,and lacks attention to the interface structure and code structure characteristics.Considering the specification of GUI interface layout based on Bootstrap framework,the relationship between components nesting form and HTML code nesting form,we proposed a different image precoding method,introduced hierarchical attention model,improved structure of the pix2 code model and extended its functionality.The main works of this thesis is as follows:(1)Propose a new image pre-training method.Autoencoder is a good way to learn important features from data.The encoder part is usually imported into other deep learning architectures to transmit the features it has learned,this is usually called migration learning.In the experiments,it's found that the training time of the model can be effectively reduced by training the automatic encoder and freezing the encoder part as the image encoder,while the accuracy rate of predicted DSL is improved.(2)Introduce the hierarchical attention mechanism.For design drawings with more components,the attention mechanism can help CNN focus on some local information of the image and generate more accurate DSL sequences.Therefore,this thesis introduces hierarchical attention mechanism,the model will subdivide the GUI drawings,then extract important features on each region,decode and combine them into one DSL sequence finally.The experimental results show that the accuracy of the DSL code generated by the model is increased after the introduction of the layered attention mechanism.(3)Improve and extend the functionality of the pix2 code model.It is necessary and valuable to generate GUI source code from hand drawings,if want to capture design inspiration,perform version iterations quickly,and further improve engineering development efficiency.The existing pix2 code can only generate DSL code segments through screenshots and render them into a single theme style GUI interface.In this thesis,the basic image data sets are transformed into an approximately hand-drawn style,besides,this thesis simplifies the HTML/CSS code in BootStrap,reduces the complexity by designing domain specific language and extends a variety of different web page styles.