DiVAS: Digital-video-audio-sketch capture, retrieval, and understanding of unstructured multimodal design knowledge

Managing and reusing knowledge can lead to greater competitive advantages and improved designs. However, design knowledge reuse often fails in the AEC (Architect, Engineering, and Construction) industry. I performed an ethnographic study focusing on one phase of the building life cycle, i.e. the shop drawing production and review process. Observations show that most of the design and detailing knowledge among the project stakeholders is typically communicated through discourse, sketch, and gesture. The knowledge created during such informal multimodal communicative events is typically lost. Even if multimedia technology is used to capture such events, the archived digital content is unstructured. Consequently, users are facing a formidable challenge to retrieve relevant content from the digital archive. This research addresses the need to capture, retrieve, and understand such digital design knowledge in the domain of shop drawing production and review.; DiVAS(TM) (Digital, Video, Audio, and Sketch), a multimedia information capture and retrieval system was developed and implemented to support these needs. A cross-media relevance model was formalized to integrate gesture-discourse-sketch digital data. This provides a macro (gesture and keywords in discourse transcript) and micro (sketch) index to a multimodal digital knowledge archive. I-Dialogue(TM) prototype was developed and implemented as a module of DiVAS(TM) system. It supports effective information retrieval over imperfect discourse transcripts. I-Dialogue(TM) provides two information retrieval algorithms that: (1) identify significant notions from imperfect speech transcripts clustered through vector analysis; and (2) disambiguate notions in the specific professional domain. I-Dialogue(TM) can be used as part of DiVAS(TM) to process speech transcripts, or as an independent system. DiVAS(TM) leverages the cross-media information retrieval process by using the macro-micro-index, which is constructed based on the formalized cross-media relevance model.; An evaluation methodology was developed to assess the recall and precision performance of DiVAS(TM) and I-Dialogue(TM). An academic test bed and a benchmark document collection from Linguistic Data Consortium were used. The evaluation results demonstrate the performance improvements achieved by DiVAS(TM) and I-Dialogue(TM) in support of effective capture, retrieval, and understanding of digital content.