Research On Clothing Aesthetics Matching Based On Deep Learning

With the improvement of living standards,people pay more attention to personal appearance,especially in terms of clothing matching.Thanks to the continuous development of deep learning and computer vision,there are many researchers conducting researches on clothing matching.They obtain information from the clothing image and corresponding text description,analyze the matching pattern of paired clothing,and then complete the task of clothing matching.Nevertheless,one important type of feature,the aesthetic feature,is seldom considered.It plays a vital role in clothing matching since a users' decision depends largely on whether the clothing is aesthetical when people decide whether a pair of upper and lower clothes suit each other.However,the conventional image features cannot portray this directly.In order to achieve the task of clothing matching aesthetically,this article introduces aesthetic information into the existing clothing matching system and proposes a new model,namely BPR-TAE-DAA(Bayesian Personalized Ranking Triple Auto Encoder Network with Domain Adaptation Aesthetic).First of all,this paper takes the aesthetic quality evaluation model NIMA,which is pre-trained on the large aesthetic visual analysis dataset AVA as the backbone network,and uses the unsupervised domain adaptive method to conduct joint training on the fashion matching dataset Fashion VC to encode the deep aesthetic information of clothing.In addition,considering that each fashion item involves multiple modalities(i.e.,visual features,text features,and aesthetic features),this article proposes to use the Triple Auto Encoder to map the multimodal features of clothing items to the same potential space.Finally,this article uses the BPR framework to explore the potential compatibility between pairs of clothing efficiently and accurately.Experiments on the fashion matching dataset Fashion VC demonstrate that the method proposed in this paper is significantly better than BPRDAE,and can recommend matching clothes that are highly related to human aesthetic perception for a given clothing item accurately and effectively.In addition,when people are observing a pair of upper and lower garments,they tend to selectively focus on certain key areas and ignore other unimportant parts.In order to simulate this selective visual attention mechanism of human beings and carry out efficient clothing aesthetic matching,this paper proposes a new clothing aesthetic matching scheme based on the Siamese network and residual attention mechanism.In this method,the Siamese network is used to extract the visual features of clothing items,and two identical residual attention networks(truncated in the sense that the FC layers are excluded)which are pre-trained on Image Net are used as the two branches of the Siamese network.Compared with the traditional backbone network Le Net in the Siamese network,the residual attention network can extract features with stronger representation ability and more consistent with human perception.At the same time,this article utilizes the bag-of-words scheme to encode contextual metadata.To seamlessly exploit the potential of both modalities in the compatibility modeling,this article first cascades visual and textual representations,and then adds K hidden layers over the concatenated vectors to integrate the visual and textual modalities of the same clothing item.Model the semantic relation between different modalities of the same fashion items.The entire network can be trained in an end-to-end fashion.Experiments have shown that the model can simulate human perception and intelligently help people recommend matching clothing items.