Research And Implementation Of Video Harmonization Algorithm Based On Style Transfer

In the era of streaming media,image and video have emerged as major carriers to support information transfer.Composition technology that extracts a specific object from one scene,and then fuses it with another scene for secondary creation,has been the research hotspot of image and video processing.However,the appearance gaps between the fused instance and background,which are caused by distinct capturing conditions,lead to an unrealistic-looking composite image or video.This thesis delves into image and video harmonization algorithms,which takes style guidance from the background and applies it to adjust the appearances of foreground objects to generate style-consistent results.The main contributions are as follows:(1)A semantic-aware visual style consistency network for image harmonization is proposed,which solve the problem that the captured overall style of background may not match the foreground content.This thesis combines a lightweight harmonization backbone with a pre-trained segmentation model,and the extracted semantic features are fed into the encoder and decoder.In addition,a spatial-separation adaptive instance normalization(SAIN)can achieve style transfer from the semantic-related background region to its foreground object.The experimental results show that the proposed method improves by 9.1%in peak signal-to-noise ratio(PSNR)compared with other methods taking the whole style guidance from background information.(2)To improve the weakening of semantic information through batch normal and other operations,this thesis also proposes an effective semantic-aware adaptive denormalization(SADE).The affine parameters that are adaptive to the input semantic features are used to modulate the convolutional feature maps,which can better preserve semantic structure information.Moreover,the aggregation strategies of this module are investigated,thereby a joint normalization structure is employed in the decoder.Experiments show that the entire network improves by 4.7%and 16.9%in PSNR and foreground mean square error(fMSE)respectively compared with other state-of-the-art methods.(3)An adaptive color-aware video harmonization method is proposed,which solve the problem that the flicker and artifact may be caused by frame-by-frame harmonization of video sequence.The harmonization network is used to predict image-specific combination coefficients of the basis look-up tables(LUTs),which records the color mappings during harmonization process.Furthermore,a keyframe scheduling strategy is introduced to pass the obtained color features of keyframes to the current frame,and a multi-scale fusion network is designed to refine the harmonization result.The experimental results show this method improves by 4.3%in time consistency metric compared with other mainstream methods.