Fashion Landmark Detection Based On Convolutional Neural Network

As a very effective visual representation of clothing image understanding,fashion landmarks are used in many applications such as clothes attribute prediction,clothes retrieval and clothes recommendation.However,due to the effects of different postures of people in clothing and non-rigid deformation of clothing,there are a large number of hard keypoints such as those which are occluded or invisible.In order to tackle this problem of hard keypoints detection,this paper proposes a targeted network model and model training method.At the same time,this paper also proposes a method that can incorporate the prior information of the clothing category to improve the detection effect.The specific content is as follows:(1)In the feature exaction level of fashion landmark detection model a novel Multi-Depth Dilated(MDD)block which is composed of different numbers of dilated convolutions in parallel and a Multi-Depth Dilated Network(MDDNet)constructed by MDD blocks are proposed.This block can effectively extract large-scale context information at different levels and hence capture more global dependencies information which is necessary for the inference of hard keypoints,and the effectiveness of the MDD block is verified through comparative experiments.At the same time,experiments show that compared with the SANL-parsing proposed in 2019,the normalization error of MDDNet is reduced from 0.0286 to 0.0251 on the dataset Deep Fashion,and it is reduced from 0.0385 to 0.0267 on the dataset FLD,achieving the state-of-the-art results.(2)To tackle the problem of hard keypoints such as those which are occluded or invisible,in the training level of fashion landmark detection model a network training method of Batch-level Online Hard Keypoint Mining(B-OHKM)is proposed.During the training of network,each clothing keypoint is one-to-one corresponding to the related loss value calculated at that keypoint.The greater the loss of the keypoint,the more difficult it is for the network to detect that keypoint.In that way,hard keypoints can be effectively mined,so that the network can be trained in a targeted manner to improve the performance of hard keypoints.In addition,the scalability of the MDD block is also discussed.Finally,through comparative experiments,the effectiveness of the B-OHKM training method and the extended MDD block are verified.The experimental results show that after the above improvements based on the MDDNet,the normalized error is reduced from 0.0251 to 0.0221 on the dataset Deep Fashion and reduced from 0.0267 to 0.0257 on the dataset FLD.(3)To improve the diversity of feature extraction for the network model of fashion landmark detection,a fashion landmark detection model that incorporates prior information of clothing types is proposed.The model uses word embedding to extract features from the prior information of the clothing category,and then fuses with the image features extracted by the convolutional neural network to form an end-to-end neural network model.Finally,the experimental on the Fashion AI dataset verify the effectiveness of incorporating the prior information of the clothing category.The above research shows that by solving the problem of hard keypoints detection and incorporating prior information into the network model,the fashion landmark detection can be greatly improved.