Multimodal interface: Auditory, haptic, and visual feedback

Multimodal interfaces offer great potential to humanize interactions with computers through the augmentation of human perceptual channels via multiple modalities. Multimodal interfaces facilitate the output as well as the input of multiple forms of information to and from the user. Due to the popularity of speech recognition technology, the majority of the multimodal interface literature, to date, has concentrated on multimodal input. As a result, there are many opportunities to contribute to the multimodal output (i.e., feedback) literature base. This dissertation provides several contributions to the literature through the investigation of a novel multimodal interface. The multimodal interface tested auditory, haptic, and visual feedback to assess how these various forms of feedback affect user workload, accuracy, and performance time. Auditory, haptic, and visual feedback were tested in several combinations: (1) alone (i.e., unimodal), (2) combinations of two (i.e., bimodal), and (3) a combination of all three (i.e., trimodal).; Thirty-two participants were asked to complete two direct manipulation tasks. Both the simple and complex tasks required the participants to perform a series of 'drag-and-drops' while the type and combination of feedback were manipulated. Performance under the different combinations of feedback was assessed with measures of mental workload, accuracy, and performance time. Mental workload was measured objectively, with the physiological measure of pupil diameter, and subjectively, with self-reported workload assessments captured by the NASA Task Load Index. The measure of accuracy recorded how precisely the participants performed the tasks. Performance time was measured by trial completion time, subtrial completion time, and target highlight time.; The results of this research demonstrate that certain types of multimodal feedback reduce mental workload and performance times, while increasing accuracy. A close examination of the combinations of unimodal, bimodal, and trimodal feedback revealed optimal feedback configurations and interface design recommendations for typical users. The findings of this research also exemplify the strength this methodological approach has for accessible design. This research serves as a baseline to drive the research and development of effective feedback combinations for individuals of varying abilities so that multimodal interfaces can enhance the performance of all users.