Dynamic Virtual Auditory Display Based On Kinect

Dynamic virtual auditory display(DVAD) is a newly developed technique for spatial sound reproduction. With a head-tracking device which can track the movement of the listener’s head, it is able to adjust the binaural synthesis processing as to adapt the temporary head position and then create the dynamic localization cue. Comparing with static virtual auditory display, DVAD produce a more realistic hearing experience.DVAD has valuable applications in scientific research, virtual reality and personal multimedia entertainment. Up to present, international and domestic research groups have developed some DVAD system, with most of them having been developed for the purpose of scientific researches and engineering applications.Although aforementioned DVAD systems achieve high reproduction performance, they require specialized head-tracking device and thus are too expensive and complicated for popular users. In addition, binaural synthesis in dynamic virtual auditory display requires a set of head-related transfer functions(HRTFs) with high spatial resolution. However, the HRTFs used in aforementioned mentioned DAVD systems are obtained by measurement. Limited by the sizes of the measurement sound source and measurement time, the spatial resolution of measured HRTFs are relatively low and spatial interpolation is needed. To address above problems, following studies are carried out in this dissertation.Firstly, HRTFs are calculated using boundary element method(BEM) and reciprocity principle. To improve the stability in calculated results, a new method of multi-source-position average is suggested. The HRTFs from KEMAR artificial head are obtained through this new method, resulting in HRTFs up to an upper frequency limit of 18 kHz and a spatial resolution of 1°.Secondly, a low-cost dynamic virtual auditory display system for personal multimedia entertainment is set up. The system includes a Kinect sensor as head-tracker, a low-latency ASIO sound card and a personal computer. The system’s software is written in C++ language and includes three different modules. These modules perform three different functions, i.e., UI control, signal processing and head tracking. In the signal processing module, a PCA-based(PCA) HRTFs binaural synthesis procedure is used. This procedure can reduce the computation cost and avoid the audible artifacts while switching HRTFs dynamically. In the head tracking modules, the system can track the listener’s face position in real time with the application of a Kinect sensor and Microsoft’s SDK.Two parameters described the dynamic performance of system, scenarios update rate and system latency time, are tested. The results indicate that the system can meet the basic need of dynamic reproduction.Finally, as a useful application, the developed system is used to improve the perceptual experiences of the binaural reproduction of 5.1 channel surround sound. A virtual source localization experiment is carried out. The results of the experiment show that with the application of dynamic reproduction system, in-head-localization and front-back confusion in headphone presentation are reduced. In addition, the dynamic system is able to improve accuracy of azimuthal localization obviously but improve little on elevation localization.