Research On Resolution Enhancement Of Complementary-structure Catadioptric Omnidirectional Imaging System

The emergency of panoramic imaging can satisfy the requirement of acquiring360o field of view of the real3D space. Due to the advantage of one-shot seamlesspanoramic imaging, catadioptric omnidirectional imaging has been widely used in manyapplications such as stereo reconstruction, robot navigation and visual surveillance.However, while looking into various applications, the problem of low and non-uniformresolution in omnidirectional imaging has become the bottleneck to its generalization.A new thought of solving the resolution problem of catadioptric omnidirectionalimaging is proposed by combining the hardware design and software processing. Firstly,a novel complementary structure of catadioptric omnidirectional imaging was designed,and then the corresponding researches on calibration of imaging system, panoramicunwrapping, image registration, image fusion and super-resolution reconstruction areconducted. The main work and innovative points are included as following:(1) Design of complementary-structure catadioptric omnidirectional systembased on multiple mirrorsWithout changing the structure of catadioptric omnidirectional imaging system, theresolution problem cannot be completely solved only depend on post processing. Acomplementary catadioptric imaging method is proposed by using multiple mirrors. Dueto the reflection of mirrors, each spatial object has two imaging positions, respectivelyin the inner and the outer of sampled omnidirectional images, which are generated fromtwo different optical paths. Two improvements of resolution have been achieved:(i)more uniform distribution of resolution,(ii) complementary resolution distributionbetween the inner and the outer. It provides basic foundation and precondition forfurther research on image fusion and super-resolution.(2) Panoramic unwrapping interpolation method of omnidirectional imagesbased on bidirectional mappingTraditional algorithms of panoramic unwrapping have not consider the distortioncharacteristics of catadioptric imaging, because the locations of neighboring points andthe distances to the current interpolated point are all measured in the space ofomnidirectional image, which seriously affects the accuracy of interpolation. Aninterpolation method for panoramic unwrapping based on bidirectional mapping isproposed. The coordinate mapping relationships are obtained by tracking optical path.Firstly, each pixel in the target cylindrical panoramic image is projected backward ontothe original omnidirectional image, and the neighboring pixels for interpolation are located. Then, all the neighboring pixels are projected forward onto the cylindricalpanoramic image, and the color value of the target pixel is interpolated according to thescattered distribution of the neighboring pixels in cylindrical panoramic images. Theproposed method avoids the adverse effect on computing the interpolation weight ofneighboring pixels caused by the nonlinear distortion, and the neighboring pixels forinterpolation are also accurately located.(3) Accurate registration of complementary panoramic images consideringpose estimation of reflecting mirrorsA pair of cylindrical panoramic images can be generated by unwrapping the innerand the outer views of complementary omnidirectional image, respectively. Accurateregistration of the paired panoramic images is the essential precondition for furtherresearch. Based on the analysis of the difference between the two panoramic images, anaccurate registration method of complementary panoramic images considering poseestimation of reflecting mirrors is proposed. Firstly, the poses of the mirrors arecalibrated based on their images of ellipse boundaries, then the inner and the outer areunwrapped into the respective panoramic images based on the calibration results; Then,more accurate sub-pixel registration of the paired panoramic images is performed byusing generalized least squares (GLS) algorithm to solve the parameters of polynomialtransform model.(4) Fusion of complementary catadioptric panoramic images based onnonsubsampled contourlet transformA novel fusion method based on nonsubsampled contourlet transform (NSCT) isproposed to fuse the pair of complementary panoramic images unwrapped from theinner and the outer of the omnidirectional image. Since the complementarity mostlyexists in the horizontal and the vertical directions, the high-frequency details areextracted by nonsubsampled directional filters (NSDFs) only in the two orthogonaldirections. The horizontal and the vertical high-frequency coefficients of the fusedimage are selected from the panoramic image which has the advantage in thecorresponding direction. The fusion rule for the low-frequency coefficients is animproved ‘selecting’ scheme with considering the high-frequency directional vector.(5) Cross-selection kernel regression method for super-resolution fusion ofcomplementary panoramic imagesThe essence of super-resolution enhancement of cylindrical panoramic space is togenerate a high-resolution panoramic image based on the scattered pixels mapped fromthe inner and the outer of the captured omnidirectional image. Adaptive kernel regression method is employed. To avoid the interference between the inner and outerimages while fusing the two complementary views, a cross-selection kernel regressionmethod is proposed. The main idea is to distinguish the two complementary viewsselectively while using kernel regression method to interpolate on the combined set ofscattered pixels.