Digital Stack Photography and Its Applications

This work centers on digital stack photography and its applications. A stack of images refer, in a broader sense, to an ensemble of associated images taken with variation in one or more than one various values in one or more parameters in system configuration or setting. An image stack captures and contains potentially more information than any of the constituent images. Digital stack photography (DST) techniques explore the rich information to render a synthesized image that oversteps the limitation in a digital camera's capabilities. This work considers in particular two basic DST problems, which had been challenging, and their applications. One is high-dynamic-range (HDR) imaging of non-stationary dynamic scenes, in which the stacked images vary in exposure conditions. The other is large scale panorama composition from multiple images. In this case, the image components are related to each other by the spatial relation among the subdomains of the same scene they covered and captured jointly. We consider the non-conventional, practical and challenge situations where the spatial overlap among the sub-images is sparse (S), irregular in geometry and imprecise from the designed geometry (I), and the captured data over the overlap zones are noisy (N) or lack of features. We refer to these conditions simply as the S.I.N. conditions.;There are common challenging issues with both problems. For example, both faced the dominant problem with image alignment for seamless and artifact-free image composition. Our solutions to the common problems are manifested differently in each of the particular problems, as a result of adaption to the specific properties in each type of image ensembles. For the exposure stack, existing alignment approaches struggled to overcome three main challenges: inconsistency in brightness, large displacement in dynamic scene and pixel saturation. We exploit solutions in the following three aspects. In the first, we introduce a model that addresses and admits changes in both geometric configurations and optical conditions, while following the traditional optical flow description. Previous models treated these two types of changes one or the other, namely, with mutual exclusions. Next, we extend the pixel-based optical flow model to a patch-based model. There are two-fold advantages. A patch has texture and local content that individual pixels fail to present. It also renders opportunities for faster processing, such as via two-scale or multiple-scale processing. The extended model is then solved efficiently with an EM-like algorithm, which is reliable in the presence of large displacement. Thirdly, we present a generative model for reducing or eliminating typical artifacts as a side effect of an inadequate alignment for clipped pixels. A patch-based texture synthesis is combined with the patch-based alignment to achieve an artifact free result.;For large-scale panorama composition under the S.I.N. conditions, we have developed an effective solution scheme that significantly reduces both processing time and artifacts. Previously existing approaches can be roughly categorized as either geometry-based composition or feature based composition. In the former approach, one relies on precise knowledge of the system geometry, by design and/or calibration. It works well with a far-away scene, in which case there is only limited variation in projective geometry among the sub-images. However, the system geometry is not invariant to physical conditions such as thermal variation, stress variation and etc.. The composition with this approach is typically done in the spatial space. The other approach is more robust to geometric and optical conditions. It works surprisingly well with feature-rich and stationary scenes, not well with the absence of recognizable features. The composition based on feature matching is typically done in the spatial gradient domain. In short, both approaches are challenged by the S.I.N. conditions. With certain snapshot data sets obtained and contributed by Brady et al, these methods either fail in composition or render images with visually disturbing artifacts. To overcome the S.I.N. conditions, we have reconciled these two approaches and made successful and complementary use of both priori and approximate information about geometric system configuration and the feature information from the image data. We also designed and developed a software architecture with careful extraction of primitive function modules that can be efficiently implemented and executed in parallel. In addition to a much faster processing speed, the resulting images are clear and sharper at the overlapping zones, without typical ghosting artifacts.