Research On Image,Video And 3D Model Segmentation And Representation

In recent years,the information exchanged in the human society becomes more colorful and contains more content.Images,videos,and 3D shapes become the next generation of information media.They often contain a large amount of information and provide straightforward representations which are very easy to understand.The rapid growth of mobile devices such as smartphones and digital cameras makes the images and videos very easy to acquire.And the development of 3D modeling technology makes 3D shapes more and more popular in our daily lives.Thus,the number of these new popular media grows exponentially.The analysis and processing of images,videos,and 3D shapes become an important problem nowadays.The segmentation and representation play important roles in the area of the anal-ysis and processing of images,videos,and 3D shapes,and have a variety of different applications.The correlation between these media provides help and guidance for the segmentation task.And the representation of these media establishes such correla-tion for further modifications and applications.In this thesis,we focus on exploring the segmentation and representation of images,videos and 3D shapes,including video object co-segmentation,video vectorization,data-driven 3D shape segmentation and indoor scene colorization,and label transfer between images and 3D shapes,demon-strating how to accurately segment those media and effectively represent them and how to apply these methods in practical applications.More specifically,this thesis mainly contains the following four aspects:(1)Video object co-segmentation via subspace clustering.The state-of-the-art meth-ods do not treat the motion information appropriately by assuming the motions between videos are consistent or different.By exploring the motion informa-tion,we propose a novel subspace clustering algorithm which combines the ap-pearance and motion information to segment the videos into consistent spatio-temporal regions.This algorithm connects the common foreground objects across different videos with the appearance features,while differentiating the foreground and background within each video with the motions.By doing so,the accuracy of the clustering is improved.Based on this algorithm,a video co-segmentation framework is established and f-ormulated as a Markov Random Field(MRF)model to optimize the final segmentation.Comparisons with previous works show the superiority of this method.(2)Video vectorization via video over-segmentation and tetrahedral representation.The state-of-the-art video vectorization remains blank while the image vectoriza-tion methods cannot be applied to videos directly due to the lack of temporal co-herence.We present a video vectorization method that generates a video in vector representation from an input video in raster representation based on video over-segmentation and tetrahedral remeshing.Based on the proposed techniques for initial segmentation,simplification,and subdivision for tetrahedral meshes,the input raster video can be converted to a sparse tetrahedral control mesh,which can be also restored to a raster video in any resolution.This method is capable of achieving high simplification ratio and ensuring color fidelity.Experiments show that the reconstructed videos of this method are faithful with tiny reconstruction errors.(3)Data-driven 3D shape segmentation and its application in indoor scene coloriza-tion.Most of the state-of-the-art 3D shape segmentation methods are based on a semantic classifier,which cannot adapt to different segmentation schemes ac-cording to different examples.So the previous indoor scene colorization method is component-based,that the annotations of the components require a lot of ef-fort.We present an image-guided mesh segmentation method to segment a 3D model into different parts according to an image object,along with a data-driven approach that colorizes 3D furniture models and indoor scenes based on it.Ac-cording to the knowledge modeled from the image-model database,this coloriza-tion framework solves an optimal reference image for each 3D model in the input scene.It is capable of transferring the colorization scheme from a reference im-age to a 3D model or an entire scene and generating a colorization scheme with a user-desired color theme.And it is also able to imitate the colorization results for those scenes containing the same type of objects,but with spatially varied pat-terns.Experiments and a user study show that our system produces perceptually convincing results comparable to those generated by interior designers.(4)Example-based image and 3D shape segmentation via label transfer.Most of the state-of-the-art label transfer methods work in only one way and the accuracy of them is not satisfactory.We present a generic and accurate method to trans-fer part-level labels from annotated 3D shapes to images or annotated images to 3D shapes.We propose a novel Local Correspondence Encoding(LCE)term to encode the probability of label assignment of each pixel as a label histogram serves as the likelihood of the labels.With the aid of local correspondence en-coding,highly accurate part-level label transfer results can be easily achieved by a Conditional random field(CRF)model and fine-scale correspondences be-tween images and 3D shapes can be established as well.This method works well for the bi-directional label transfer between 3D shapes and images,suitable to image-guided 3D shape segmentation,3D shape-guided image segmentation,and has also found applications in image-based mesh colorization,image depth estimation,and 2D sketch segmentation.We thoroughly evaluate our method and demonstrate its superiority over the state-of-the-art methods through experi-ments on benchmark datasets.