Research On Fashion Compatibility Modeling With Neural Networks

In recent years,there has been a growing interest in the fashion analysis(e.g.,compatibility modeling of fashion items)due to the huge economic value of fash-ion industry.Clothing is indispensable in people’s daily life.With the continuous innovation of clothing style,pattern and material,it plays an increasingly impor-tant role in daily life.In addition to meeting the great demand of people for clothes,more and more people begin to pursue the fashion,generosity and decen-cy.Nevertheless,users find it difficult to match the complementary fashion items and make proper outfits,facing the huge amount of fashion items.Consequent-ly,it deserves our attention to develop an effective clothing matching scheme to help people figure out the suitable match for a given set of fashion items and make a harmonious outfit.As a matter of fact,an outfit usually consists of multiple complementary items(e.g.,a top,a bottom and a pair of shoes),the key to a suitable outfit is the compatibility among the complementary fashion items.As deep learning technology has achieved remarkable success in the field of representation learning,it has become the main technical approach of existing compatibility modeling methods.In view of the extensive research interests of fashion analysis have received in the computer vision,information retrieval and multimedia communities,and the actual needs of users for compatibility analysis of fashion items,we mainly study how to effectively develop the multi-modalities and effectively learn the feature representation of fashion items,how to explore the influence of different attributes of fashion items on compatibility modeling and how to capture the compatibility relationship between complementary fash-ion items from different perspectives,by designing more effective compatibility modeling methods to enhance the overall performance.Under the support of the National Natural Science Foundation of China,we focus on the compatibility modeling task in recommendation system.It mainly studies the problem of clothing modality lacking and sparsity relationship among fashion items,attributes association of different categories of fashion items and the complex compatibility modeling of complementary fashion items.The main research contents and innovations of this dissertation can be summarized as fol-lows:(1)We propose a multi-model compatibility modeling for complementary fash-ion items based on neural networks.The traditional compatibility modeling meth-ods mainly leverage the visual modality of fashion items and ignore the textual metadata.To comprehensively model the compatibility between fashion items,we mine the intrinsic relatedness between the textual and visual modalities and the influence of multi-modal information on compatibility modeling.Firstly,we present a multiple autoencoder neural network,which is able to not only model the compatibility between fashion items(e.g.,a top and a bottom,a bottom and a pair of shoes),but also accomplish the compatibility modeling for multiple fashion items by seamlessly exploring the multi-modalities(i.e.,the visual and textual modalities)of fashion items.Considering that the compatibility among different fashion items(e.g.,tops,bottoms and shoes)can be rather complex,which is usually affected by many factors such as the color,shape and functional-ity,and the sparse relationship among fashion items.To accurately measure the compatibility between fashion items,we further propose a content-based neural scheme,which aims to learn the latent compatibility space and bridge the seman-tic gap among fashion items from heterogeneous spaces.Meanwhile,in order to make the utmost of the implicit feedback pertaining to the compatibility among fashion items,we adopt the bayesian personalized ranking framework to explore the matching preferences among the complementary fashion items.Finally,we propose a multiple auto-encoder neural network based on the bayesian person-alized ranking scheme,which is able to jointly model the implicit preferences among fashion items and the relationship between different modalities of them.We construct a fashion dataset FashionVC+,consisting of both the visual and textual metadata of fashion items(i.e.,the tops,bottoms and shoes).The ex-perimental results show that the proposed scheme is able to jointly model the coherent relationship between different modalities of fashion items and the im-plicit preference between them,and verify the advantages of considering textual modality in compatibility modeling.(2)We propose an end-to-end attention-based compatibility modeling method.Given a top and a bottom along with their visual and textual metadata,we aim to effectively learn the latent multi-modal representations of fashion items,which would enable us to accurately capture the various aspects(e.g.,patterns,color and categories)and then measure the compatibility between them.Different from the previous works that treat all features equally,we further consider the different contributions of different features(i.e.,the high-level features extracted from neural networks)to compatibility modeling.In particular,we design an end-to-end multi-modal deep neural network,which is capable of effectively learn the feature encoding of multi-modalities and model the compatibility preference between complementary categories of fashion items.Furthermore,to distinguish the contributions of different pair-wise features in the compatibility modeling,we propose a feature-level attention model to adaptively assign the confidence of different features for different fashion items.The results show that the proposed feature-level attention model can effectively learn the corresponding confidence for the pair-wise features and thus enhance the performance of compatibility modeling.In addition,we also get some interesting insights.For example,the distributions of the feature confidence for the similar bottoms candidates are similar,while the bottoms of different categories vary greatly.(3)We propose a multi-modal generative compatibility modeling method.In a sense,existing compatibility modeling methods mainly focus on learning the latent compatibility space with advanced neural networks to bridge the gap be-tween complementary fashion items,where the item-item compatibility can be directly measured based on the representation of each fashion items that learned in this space.As a matter of fact,in addition to exploring the potential compat-ibility space,the compatibility rules between complementary fashion items can also be generated directly,that is,the compatibility rules between products can be described with the complementary compatibility template.This dissertation aims to sketch a compatible bottom template for the given top to describe the underlying compatibility(features),which can as a reference and hence facilitate the final compatibility modeling between the top and the bottom.Firstly,we in-troduce an auxiliary complementary template generation network equipped with the pixel-wise consistency and compatible template regularization,working on transferring the multi-modalities of the given top to a compatible bottom tem-plate.Then,based on this template,we propose an auxiliary template-enhanced generative compatibility modeling scheme,which is able to seamlessly simultane-ously integrates the primary item-item compatibility modeling and the auxiliary item-template compatibility modeling.The experimental results show the ad-vantage of considering the item-template compatibility.Although the generated templates can only capture the item shape and color well rather than the texture,the performance of compatibility modeling can be significantly boosted.