A Study Of Synergy Between Modalities In Dynamic Multimodal Discourse

With the use of computer and multimedia technology, mankind has entered the colorful multidimensional digital information age of fast popularization. Discourse has been no longer in the traditional written and spoken forms, but has developed to a multimodal one with co-existence of combining image, language and audio, etc. As a bran-new field at home, MDA has been paid more and more attention. Most of the MDA is mainly based on Kress & van Leeuwen s VG(2006) as theoretical basis concentrating on the study of static multimodal discourse, and some research focus on dynamic multimodal discourse studying the modal relations in classroom and movie.With Kress & van Leeuwen s VG(2006), this paper studies the synergetic effect of PSA modality in the aspect of ideational function, interpersonal function and textual function based on Halliday s(2004) SFG. This study takes the most welcome CCTV PSA-- Taking Care of the Dementia Elder People as a sample to do the research and captures 16 shorts of visual and auditory mode for analyzing by using the software Adobe Audition CS6 to transform audio text and Edius 6 to make video text.Through the analysis, this paper has found that: in the perspective of ideational function, verbal modality and visual modality are mainly in complementary relationship to intuitively reflect the interactive relationship between the participants and image viewers, leave the image viewers deep impression of lonely inner world of dementia elder people and reveal the viewers common thoughts and feelings. Thus, it appeals for filial piety and caring for dementia elder people in the whole society. However, the non-complementary relationship between verbal modality and visual modality reflects the interaction of participants showing the daily behavior of dementia elder people through overlapping and implying.In the perspective of interpersonal function, the synergetic effect between verbal modality and visual modality is completely complementary to strengthen each other. Mode of voice and interpersonal system in verbal modality show the distance between the viewers and the participants and let the viewers participate in the story involving son and father to realize the deep love that the father expresses to the son. Visual modality shows the authenticity of the images using high modal to establish a close relationship between the participants and the viewers, then the viewers could naturally integrate into participants world.In the perspective of textual function, tonic prominence in audio modality and information value in visual modality have a direct complementary or non-complementary relationship. Information maximum value highlights the weak position of marginal elements in PSA, so as to arouse the viewers mercy and compassion to dementia elder people.In this paper, the research shows that the modality synergy in dynamic multimodal PSA is an important way to present audience a real, contagious world.