Research On Optimal Seamline Detection And Seamline Network Generation For Orthophoto Mosaicking

While mosaicking multiple orthoimages,especially captured from the urban scenes with many obvious objects,the detection of an optimal seamline in an overlapped region and the generation of a seamline network are two key issues for creating a seamless and pleasant large-scale digital orthophoto map(DOM).In order to avoid the appearance of artifacts caused by geometric misalignments along seamlines,the detected optimal seamlines should avoid crossing obvious objects,e.g.,buildings or moving cars,and should cross regions with high image similarities and smooth textures,e.g.,roads,rivers and grasses.For multiple orthoimages mosaicking,the seamline network should be constructed to connect the individual seamlines with the optimal junction points.The traditional pixel-based optimal seamline detection approaches are easy to be completed and efficient.However,it is difficult to distinguish obvious objects from other contents in orthoimages using only the image difference defined at the pixel level.Therefore,it is important to propose the new object-based seamline detection approaches.In addition,most of approaches detect the seamlines at pixel level.It can detect high-quality seamlines,but it is time-consuming because the number of nodes is large.Therefore,the new optimization strategy should be proposed to improve the efficiency.The seamline detection approaches focus on how to find the optimal seamline between two images and cannot handle multiple images.However,in most cases,we need to stitch multiple orthoimages to generate the final large-scale DOM.The seamline network should be constructed to handle this problem.The key problem of seamline network generation is how to automatically find the optimal junction nodes.However,relatively few methods have been proposed to solve this problem.The main research work and innovative achievements are as follows:(1)A novel foreground segmentation-based seamline detection approach and a coarse-to-fine seamline optimization strategy are proposed in this paper.The pixel-based seamline detection approach cannot avoid the seamline crossing the obvious objects in some cases.We find that most of the regions which are unsuitable to be crossed by the seamline are the foreground regions of two input images.Therefore,we propose a novel foreground segmentation-based approach to detect the optimal seamline for two adjacent images.First,the foreground objects are segmented from the overlapped region between two images at superpixel level.Then,the energy cost map generated by using normalized cross correlation is enhanced on the regions of foreground objects.Last,the optimal seamline is detected from the energy cost map by using graph cuts.This approach has an essential advantage in avoiding the crossing of obvious objects.In addition,we also propose an effective and efficient strategy that minimizes the energy functions represented by the difference between two images via graph cuts.This strategy first detects the optimal seamline via graph cuts at superpixel level and then refines it at pixel level,and is capable of improving the efficiency of energy optimization and does not affect the quality of the detected seamline.(2)A novel semantic segmentation-based seamline detection approach is proposed in this paper.In some cases,the foreground segmentation-based seamline detection approach cannot avoid the seamline crossing the small objects.Therefore,we propose a new approach to detect optimal seamline for orthoimage mosaicking with the use of semantic segmentation completed using a deep convolutional neural network(CNN).We first propose a deep CNN for land cover semantic segmentation in overlap regions between two adjacent images.Then,the energy cost of each pixel in the overlap regions is defined based on the classification probabilities of belonging to each of the specified classes.To find the optimal seamlines globally,we fuse the CNN-classified energy costs of all pixels into the graph cuts energy minimization framework.The main advantage of our proposed method is that the pixel similarity energy costs between two images are defined using the classification results of the CNN based semantic segmentation instead of using the image information of color,gradient or texture as traditional methods do.Another advantage of our proposed method is that the semantic information is fully used to guide the process of optimal seamline detection,which is more reasonable than only using the hand designed features defined to represent the image differences.(3)A novel seamline network construction and optimization approach is proposed in this paper.We propose a novel seamline network generation approach to produce large-scale DOM by mosaicking multiple orthoimages.We first generate the initial seamline network by assigning each pixel in image mosaic to the nearest image center.Then,the pairwise and junction regions extracted from the initial network are refined using two-label and multi-label graph cuts,respectively.The key advantage of our proposed seamline network is that junction points can be automatically and optimally found using the multi-label graph cuts.In addition,the proposed approach can generate high-quality seamline network with less artifacts and regular effective mosaic polygons.The experiments are conducted on several groups of orthoimages captured from different urban areas to evaluate our proposed seamline detection approaches,coarse-to-fine energy optimization strategy,and the seamline network generation approach.The experimental results show that our proposed approaches are effective and superior,outperform the state-of-the-art approaches and the commercial software based on visual comparison and statistical evaluation.Our proposed approach can produce high-quality seamlines and seamline networks with regular effective mosaic polygons and less artifacts.