Scalable Optimization Methods with Side Information in Image Understandin

The field of Computer Vision includes a highly varied collection of technologies for using Artificial Intelligence to understand and process images. A common thread throughout Computer Vision is mathematical optimization, frequently used as a tool to model and solve these problems. A key advantage of optimization formulations is that a model of any basic Computer Vision problem can be extended to include additional information and requirements. Prior knowledge, side information, and application-specific restrictions can be expressed within the objective or constraints. This provides a general way to formally pose modifications to common vision algorithms. It is not entirely sufficient, however, to only describe an optimization model that includes the desired side information. The wide array of problems that can be formulated this way is accompanied by a similarly wide range of computational difficulty. While the optimization problem for a standard Computer Vision problem may be solved by a simple and efficient algorithm, with included side information the extended problem can be fundamentally harder and require more complex solvers.;The focus of this dissertation is a set of vision applications in image segmentation, clustering, and classification that include side information. For the extended problems, I describe scalable and distributed algorithms that allow even the harder optimizations to be solved efficiently. In the case of segmentation, the inference is extended to consider multiple images related by the presence of a common foreground, with an interactive implementation that can parallelize across the computational units of a Graphics Processing Unit (GPU). Then, a distributed image clustering algorithm that can incorporate side constraints is presented. The final problem that is considered is the use of side constraints in neural network training to build image classifiers with reduced memory requirements. This dissertation shows that modeling Computer Vision problems as an optimization effectively provides a way both to reason about the kinds application-specific extensions presented in these examples and to make finding a solution fast and efficient.