Distributed multi-modal human activity analysis: From algorithms to systems

Human activity analysis is one of the most important and challenging research fields of information technology. It has a significant impact on applications such as semantic multimedia indexing, human-machine interaction, and homeland security. This dissertation discusses the search for the most effective and efficient architecture for human activity analysis from both an algorithm and system perspective. The discussion starts from the most fundamental system setup that of single uni-modal sensor processing. Afterwards, to enhance the performance from single uni-modal sensor processing, two architectures are discussed, distributed uni-modal sensor processing and single multi-modal sensor processing. Distributed uni-modal sensor processing applies multiple uni-modal sensors in a network to increase the range of the sensing views, and to decrease the sensitivity to environmental changes. On the other hand, applying a single multi-modal sensor extends the scope of analysis by having more sensing modalities. Combining these two directions, a distributed multi-modal human activity analysis system is proposed. This architecture demonstrates a significant advantage in its effectiveness of human activity analysis. However, managing such a complicated system is challenging, so a dedicated monitoring framework is proposed at the end of this dissertation to resolve this system design issue. With the promising system proposed by this dissertation, human activity analysis can further advance to another level.