Research On Key Technologies Of CMOS With Rolling Shutter In Aviation Application

Since solid-state image sensors replaced films as the main imaging equipment, Charge Coupled Devices(CCD) have seen their absolute dominance in the imaging device market, especially in the field of aerial camera. In recent years, more and more attention has been paid on CMOS, because it has multiple advantages, such as low cost, fast speed and integrated image processing unit on chips. Meanwhile, the image quality of CMOS is also improved significantly. Its application is no longer confined to the consumer electronics market, but began to expand in the field of aerial imaging. Different from CCD with global shutter, CMOS often adopts electronic rolling shutter(RS). Each row of a CMOS is exposed at different time intervals and is readout sequentially. Therefore, when there is a fast relative motion between the CMOS aerial camera and the object scene, final images not only suffer from motion blur, but also have geometric distortions caused by RS effect. Furthermore, this problem is exacerbated when the gesture angles(pitch, roll, and yaw) of the aerial camera change. In this situation, since the velocity of each pixel is closely related to the pixel’s location, the distortion of the whole pixel array is exceedingly complicated. In order to expand the application of CMOS in the field of aerial camera, the research on the imaging problem of high-speed CMOS is very important.This dissertation firstly introduces the differences between CMOS and CCD, analyzes the principle of RS effect in detail and then proposes a theoretical model of RS distortion in motion imaging. Moreover, we deeply research on the traditional algorithms of removing RS effect and put forward a method of removing RS distortion based on optical flow. Through analyzing the imaging model of aerial camera, we establish a distortion model of CMOS when imaging at arbitrary gesture angles. Based on this model a distortion correction method of single CMOS image is proposed. In order to solve the problem of image blur and geometric distortion caused by high-speed CMOS aerial cameras, we establish a mixed model and propose a restoration method of double degraded images based on estimation of blur parameters.The main innovations and research results are as following:1、First the basic structures of CMOS and CCD are introduced in detail. We discuss the connections and differences between them in all aspects. Based on the differences in the imaging principle, the problem of RS effect is derived. Then, by analyzing the movement of each pixel in high-speed CMOS imaging, we quantitatively analyze RS distortions in different directions and establish the reference model of RS effect. Finally, we summarize and analyze the traditional algorithms of removing RS distortions and outline their steps, which provides a guideline for subsequent distortion correction methods.2、In order to solve the problem of low efficiency of traditional methods which cannot meet the requirements of real-time aerial imaging, a novel approach is proposed to correct RS distortions based on the inter-frame motion estimation by optical flow. We put forward a simplified affine model according to the established RS effect model and through estimating the motion between two adjacent video frames by using the optical flow method, the parameters of affine model are calculated. Finally the restored images are obtained by reverse projection. The proposed method simplifies the calculation process, which makes the calculation of parameters become fast and greatly improves the efficiency.3、According to the imaging model of aerial camera, the velocity of each pixel is calculated through the method of coordinate transformations. Then, the imaging distortion model of CMOS aerial cameras at arbitrary gesture angles is established. Based on this, a distortion correction method of single CMOS image is proposed. It can directly calculate the distortion map of the whole distorted image according to the motion information of aerial camera and can finally conduct the restored images by affine transform. Error analysis based on the Monte-Carlo statistical method indicates that the accuracy of this model fully meets the requirements of aerial imaging. And this method directly calculates the distribution of CMOS distortions, which skips the complicated inter-frame motion estimation. Therefore, the computation time of this method is significantly reduced, which makes it meet the needs of aerial real-time imaging. The experiment results demonstrate the effectiveness of our method.4、To address the problem of image motion blur and RS distortion caused by high speed CMOS cameras, a novel idea of unifying this two kinds of image degradation factors is proposed. By analyzing the respective mathematical model and taking the motion vector as a bridge, we establish a hybrid model of two factors and propose an algorithm of dual degradation image restoration based on parameter analysis. In this method, the motion parameters of the camera are estimated first by the commonly used blur parameter estimation method, and then the motion blur and the RS distortion are recovered by the estimated parameters. Under the condition of ensuring the recovery effect, just one-time estimation can eliminate the influence of two factors, which can simplify the process and can significantly improve the efficiency. This method provides a new way to address the problem of real-time imaging in CMOS aerial cameras.