Researches Of Digital Image And Video Inpainting Algorithms

The digital image or video inpainting technique aims to fill missing pixels in unknown regions of an image or video in visually plausible way by utilizing the information from the un-missing region. Different from the image restoration, the digital inpainting technique does not request the result to reappear the original image. Its objective is to conceal the missing or damage parts of the images or videos and restore it in a unity way that is non-detectable for an observer who does not know the original image. Along with the popularity of the mobile image and video acquisition devices, the requisite for the advanced image processing technique of people is increasing dramatical. Compared with the image restoration, image enhancement and image denoising technology, the inpainting technology possesses high operating freedom and it becomes a high-profile technology in the digital image processing field. Digital image inpainting is not only an emerging technique, but also a hot issue in the image processing and computer vision research. With the development of inpainting technique, it has found broad applications in heritage conservation, photo restoration, special effects, and errors conceal in videos, disocclusion in computer in computer vision and so on. However, due to a wide variety of images and videos and missing regions, the digital inpainting methods are also varied.This dissertation attempts to research on image inpainting and video inpainting techniques as well as their applications, according to the analysis of the traditional inpainting algorithm and the existed image processing theory. In this dissertation, a series of methods have been proposed for the detection of caption text in image and video, the small textural missing region restore of image, the large scale missing completion of image and video inpainting problem. The main contributions of this dissertation are as follows:1. The research of superimposed text detection in images and videosToday, more superimposed texts are embedded within images and videos. Usually some texts are unnecessary. Thus, many applications require an approach to remove the text and complete the video. Moreover, the current image and video restoration methods need to specify the inpainting target area by the human intervention. The manually marking of the superimposed text at the image or video is not only time-consuming but also unreliable. To automatically locate the superimposed text and provide the target region for the inpainting algorithm, this dissertation proposes a classification-based algorithm for text detection using a sparse representation with discriminative dictionaries. First, the edges are detected by the wavelet transform and scanned into patches by a sliding window. Then, candidate text areas are obtained by applying a classification procedure using two learned discriminative dictionaries. Finally, the adaptive run-length smoothing algorithm and projection profile analysis are used to further refine the candidate text areas. The proposed method is evaluated on a large number of images and video frames. The various experiments shows that the proposed text detection method not only accurately extracts the artificial subtitles of photos and videos as target area for the image and video inpainting but also allows robust text detection.2. The research of small scale textural missing image inpaintingSparse representation is a theory and research focus in the image processing and signal processing. It is a novel signal representation theory in succession to the multiresolution transforms such as wavelet and curvelet. Compared to the traditional multiresolution transforms, the sparse representation is closer to the human visual characteristics. In the image inpainting field, this dissertation first proposes an improved sparse representation inpainting approach based on source image dictionary according to the framework of the classical sparse representation inpainting method. By considering the defects of predefined or learned dictionary sparse representation inpainting, this dissertation use a source image dictionary to replace the traditional dictionary, which makes sparse representation inpainting approach more effective. Then to handle the small scale textural missing image inpainting problem, a effective inpainting method is proposed, which combines fast inpainting method and source image dictionary based sparse representation inpainting method. A texture distribution analysis algorithm divides the missing area into homogeneous region and inhomogeneous region. Then the fast inpainting method restores the homogeneous region and sparse representation inpainting method recovers the inhomogeneous region. The proposed method fully considers the complementary between the fast inpainting method and sparse representation inpainting approach. In this manner, the hybrid inpainting method inpaints the small size textural missing image more effectively than the available inpainting method.3. The research of large scale missing image inpaintingIn the large scale missing image inpainting field, ensuring the integrity and consistency of damaged structure is the key to obtain good results of inpainting. At the beginning of Chapters4, this dissertation analyzes the defect of existing texture synthesis based image inpainting method and the human visual perception in image missing detection and filling. Then the idea of large scale inpainting method is proposed.Inspired by human visual characteristics, a new image inpainting approach which includes salient structure completion and texture propagation is introduced. In the salient structure completion step, incomplete salient structures are detected using wavelet transform, and completion order is determined through color texture and curvature features around the incomplete salient structures. Afterwards, curve fitting and extension are used to complete the incomplete salient structures. In the texture propagation step, the completed salient structures divide the target area into several sub-regions. The texture propagation is used to synthesize the texture information with samples from the corresponding adjacent sub-regions. This reduces the running-time and offers more precise texture information. A number of examples on real and synthetic images demonstrate the effectiveness of our algorithm in removing occluding objects. The experimental results compare favorably to those obtained by existing inpainting techniques.4. The research of video inpainting methodVideo inpainting is a continuation of the image inpainting technique. It is an important part and application of inpainting technique. To deal with the main issues of video inpainting:fixed information removing and motion object repairing, this dissertation proposes two different video inpainting methods. One is the video inpainting based on three-dimensional Poisson equation to remove the fixed information of videos. The other is foreground replacement based video inpainting via motion cycle detection method to handle the motion object repairing problem.In the three-dimensional Poisson equation video inpainting method, firstly, the gradient fields of video frame are extracted. Then, the target area of every gradient field is filled through patch-by-patch inpainting method. Afterwards, to repair the video, a three-dimensional Poisson equation is built and solved in the gradient field of the target area. The experimental results show that the three-dimensional Poisson equation video inpainting method can achieve desired results and perform better than existed methods in the colors, light and shade consistency.In the motion object repairing issue, the challenge of video inpainting is how to complete the damaged moving foreground objects in the spatiotemporal domain. Therefore, this dissertation presents a novel foreground-background inpainting method for completing missing information based on object motion analysis. In the foreground replacement based video inpainting via motion cycle detection method, the foreground in the video is firstly separated from the background. As for background inpainting, we use spatiotemporal copy method. In the foreground inpainting stage, a motion cycle of the moving object is detected using skeleton similarity. After that the damaged foreground object is directly replaced by corresponding undamaged foreground in the motion cycle. The proposed method is useful for a variety of tasks, including, static and translation camera motion and large object movement, moving target region and variation background.Finally, the dissertation summarizes the main contribution, innovative research achievements and the future work.