Researches On The Removal And Application Of Shadows In Images

Shadow is a common optical phenomenon.When light is blocked in natural scenes,insufficient illumination on the object’s surface leads to the formation of shadows.The research and application of shadows are meanstream and important research fields in computer vision.On one hand,shadows are a kind of noise for high-level computer vision tasks,shadow removal studies can support these tasks.On the other hand,shadows contain rich information on scenes and light sources.How to use shadow images for effective scenarios information extraction is vital for scene understanding and other vision-based applications.Therefore,researches on the removal and application of shadows have important theoretical significance and application value.In this paper,we focus on the research of shadow images.Firstly,we study the fully-supervised shadow removal method based on the intrinsic characteristics of the shadow image from the perspective of shadow formation.Secondly,to solve the data limitation problem of the fully-supervised shadow removal methods,we study the lightness-guided shadow removal method with training on unpaired data based on the current unpaired shadow removal methods.Thirdly,due to the shadow-free images are still hard to capturing,we study a shadow-generation-based weaklysupervised shadow removal method.Finally,inspired by shapes,orientations and other characteristics of shadows,we explore a new application of shadow images–shadow vanishing point detection via human/shadow,where human/shadow represents the combined region of the human and its shadows.We study the related methods,theories,and applications thoroughly,and the main contributions of this paper are as follows:(1)An intrinsic-characteristics-based method for fully supervised shadow removal is proposed.The current methods based on shadow formation model cannot properly process complex shadows as most of them simplified the model to perform shadow removal.To address this problem,a novel shadow image formation model is proposed.In the shadow image formation model,we represent each shadow-free pixel as a transition function of illumination parameters,reflectance parameters,and the shadow pixels.We present a new shadow removal method LR-Shadow Net(Shadow Removal via Estimating Lightness and Reflectance)based on the proposed model.LR-Shadow Net consists of two sub-nets for parameter estimation and one sub-nets for result refinement.Parameter estimation sub-nets are used to estimate the illumination and reflectance parameters,and we propose to use the physical properties for obtaining accurate parameters.Result refinement sub-net can solve the deviation of parameters caused by the lack of supervision information.Several spatial feature transform blocks are incorporated into LR-Shadow Net and they can help guide parameter estimation and result refinement based on shadow region information.Extensive experiments on the widely used AISTD and VSRD datasets show that the proposed LR-Shadow Net achieves better performance with stronger generalisation ability than current shadow removal methods,and LR-Shadow Net can process more complicated shadows.(2)A new lightness-guided method for shadow removal by training on unpaired data is proposed.Current methods trained on unpaired data have a large number of parameters to be learned and are hard to converge and cannot accurately remove shadows,resulting in worse shadow-removal performance.In this method,we solve the above problems by presenting a Lightness-Guided Shadow Removal Network(LG-Shadow Net).LG-Shadow Net consists of a lightness-guided module and a shadow removal module.The lightness-guided module aims to compensate for the lightness of the shadow region,and the shadow removal module aims to learn the whole shadow knowledge for shadow removal.for better use of the data prior,we introduce a loss function to further utilise the colour prior for improving the shadow removal performance.Extensive experiments on widely used ISTD,AISTD,and USR datasets show that LG-Shadow Net performs better than the other methods with trained on unpaired data.Besides,LG-Shadow Net achieves better results than previous unpaired methods and has fewer parameters and higher computational efficiency.(3)A shadow-generation-based weakly-supervised shadow removal method is proposed.The performance of current methods is restricted by the diversity of shadow-free data.We solve this problem by proposing a new G2R-Shadow Net that leverages shadow generation for weakly supervise shadow removal by only using a set of shadow images and their corresponding shadow masks for training.G2 RShadow Net first builds a shadow generation sub-net which stylises non-shadow regions to be shadow ones,leading to paired data for training the shadow-removal sub-net.The third sub-net is built for adjusting the colour of the shadow removal region to be consistent with the other regions based on the context information of the shadow region.Extensive experiments on AISTD and VSRD datasets demonstrate that the proposed method can learn shadow removal without the shadow-free images.And it achieves better performance and generalisation ability than the current weakly supervised method,which proves the effectiveness of the proposed method.(4)A human/shadow-adaptive feature extraction method for shadow vanishing point detection is proposed.This is the first method that explores the utilise of human/shadow images for detecting vanishing points.We propose a deep-learningbased Shadow Vanishing point Detection Network(Shadow VPNet).In Shadow VPNet,a human/shadow detector is first built to obtain the human/shadow masks.Then a human/shadow adaptive feature extractor is proposed to extract the human/shadow features with the guide of the human/shadow mask.Finally,a vanishing point classifier is built to predict the final shadow vanishing point with the human/shadow features and candidate shadow vanishing points as inputs.We construct a new Shadow Vanishing Point Image dataset(SVPI),as well as all the needed ground-truth annotations.Experiments on the proposed SVPI demonstrate that using the human/shadow in shadow images can help to accurately detect the shadow vanishing point,and the proposed method achieves better performance and generalisation ability than current methods,and the experiments in argument reality further proves the effectiveness and promising application prospects of Shadow VPNet.