Hand Tracking, Finger Identification, And Chordic Manipulation On A Multi-touch Surface

This research introduces methods for tracking and identifying multiple finger and palm contacts as hands approach, touch, and slide across a proximity-sensing multi-touch surface (MTS). Though MTS proximity images exhibit special topological characteristics such as absence of background clutter, techniques such as bootstrapping from hand-position estimates are necessary to overcome the invisibility of structures linking fingertips to palms. Context-dependent segmentation of each proximity image constructs and parameterizes pixel groups corresponding to each distinguishable surface contact. Path-tracking links across successive images those groups which correspond to the same hand part, reliably detecting touchdown and liftoff of individual fingers. Combinatorial optimization algorithms use biomechanical constraints and anatomical features to associate each contact's path with a particular fingertip, thumb, or palm of either hand. Assignment of contacts to a ring of hand part attractor points using a squared-distance cost metric eectively sorts the contact identities with respect to the ring structure. Despite the ascension of the mouse into everyday computing, more advanced devices for bimanual and high degree-of-freedom (DOF) manipulation have failed to enter the mainstream due to awkward integration with text entry devices. This work introduces a novel input integration technique which reserves synchronized motions of multiple fingers on the MTS for multi-DOF gestures and hand resting, leaving asynchronous single finger taps on the MTS to be recognized as typing on a QWERTY key layout. The operator can then switch instantaneously between typing and several 4-DOF graphical manipulation channels with a simple change in hand conguration. This integration technique depends upon reliable detection of synchronized finger touches, extraction of independent hand translation, scaling, and rotational velocities, and the aforementioned finger and hand identications. The MTS optimizes ergonomics by eliminating redundant pointing and homing motions, minimizing device activation force without removing support for resting hands, and distributing tasks evenly over muscles in both hands. Based upon my daily use of a prototype to prepare this document, I have found that the MTS system as a whole is nearly as reliable, much more efficient, and much less fatiguing than the typical mouse-keyboard combination.