A Study Of The Synergy Between Modalities In Multimodal Discourse

Since1950s, the research on discourse analysis has been developed for more than half a century. During this period, the related studies mainly focus on the traditional perspective, linguistic discourse. However, with the development of science and technology, people actually communicate in various ways besides language, just to name a few, gesture, gaze, posture, motion, sound, tone of voice, music, three-dimension objects, tables, diagrams and pictures. Therefore, the research on multimodal discourse analysis, which consists of various resources, is of increasing importance as a new research field. These studies mainly focus on the perfection of theoretical framework to analyze multimodal discourse and its application and only a few refer to the mechanism different modalities interact with each other. However, the research on the synergy between modalities contributes a lot to the establishment of the production and interpretation mechanism for multimodal discourse. Therefore, the researches on this aspect are in demand.This study is based on the research materials, five issues of the television interview program, One on One, shown in CCTV news on Sundays. The software, Praat, is used to transcribe the auditory modality and the software, Sony Vegas is utilized for the transcription of the visual modality. And then, the Systemic Functional Grammar is employed to analyze the text of each material, especially the transitivity system; for the vocal perspective, the pitch is used to reflect mood and the tonic prominence, intensity and pitch contour, indicates the information focus. For the study of the visual modality, Grammar of Visual Design is the theoretical basis for the analysis of images. The analysis is conducted from three aspects, i.e. representational meaning, interactive meaning and the meaning of composition. Eventually, the synergy between visual and auditory modality is observed from three aspects, i.e. three meta-functions, ideational function, interpersonal function and textual function.As a research on this topic, figuring out what is multimodal discourse is the primary purpose. Therefore, the thesis defines multimodal discourse in two levels:the one in broad sense and the one in narrow sense. For the synergy between auditory and visual modality, in the realization of the ideational function in multimodal discourse, there are complementary and non-complementary relations between modalities. The former is further divided into intensive and non-intensive relations, and the latter, interactive and entailment relation. In the perspective of interpersonal function, the synergy between auditory modality and visual modality is complementary intensive relation in the whole. In the realization of the textual function, the synergy is classified into four types, i.e. complementary intensive relation, complementary non-intensive relation, non-complementary interactive relation and non-complementary entailment relation.