Fingertip-based Interactive Projector-camera System

With projectors becoming smaller and more affordable, projection displays are no longerlimited to traditional entertainment functions such as showing movies, playing video games, etc.They are now also being used for education, visualization, simulation, design and interactionapplications. During the past few years, great progress has been made in computer visionalgorithms, leading to higher flexibility and more natural interaction of bare hand interfaces overtangible interfaces.This paper aims at building a human computer interaction system comprising a projectorand camera, where a regular projected surface functions as a touchscreen, and user can interactwith the computer by using the fingertip. A novel algorithm is proposed, which robustly detectsfingertip and archives touch-operated controls with a high precision of few millimeters.For vision-based human-computer interaction, extraction of user region from cameraimages is an essential part of image processing. We present a simple, accurate and robust methodto segment the dynamic background and extract user region, based on the fact that theappearance of the projected image in camera images is inherently predictable from the projectorinput images. The prediction is done by formulating the geometric and photometrictransformation between the projector and the camera. Therefore, we can extract user regionprecisely by the difference between the captured image and the predicted image. We firstcompute the predicted image and then distinguish the user region from the background byanalyzing the changes of surface albedo. After hand region extraction we employ curvature forthe segmentation of fingertip.Also, we can easily detect the fingertip using curvature methods. The challenge lies in thatfrom a single image alone it is difficult to detect whether there is a physical contact between thefingertip and the display surface. The facility of acquiring3D information about the projectorcamera system would be of much aid. A desirable way of making that possible is using thetriangulation. This way, the projector serves as the display device, as well as the3D sensor at thesame time. The experiment shows good results and good performance.We have two important contributions: firstly, we develop a simple and robust hand detectionmethod which can work well in a wide variety of situations including dark and luminous ambientillumination. Secondly, a camera and a projector comprise a stereo vision system. We determinewhether a physical touch takes places by triangulation.