A Multimodal Study On Experiencing English Viewing Listening & Speaking From The Perspectives Of Visual Grammar And Intersemiotic Complementary Theory

Multimodal discourse analysis is an approach to analyzing the interrelationship between different modes in creating a unified text based on Systemic Functional Linguistics. Multimodality has been one characteristic of textbook discourse now, and more and more efforts are devoted by scholars to study multimodal features of the textbooks. Even though many researchers have conducted discourse analyses on textbook discourse from multimodal perspectives, they mainly focus on scientific textbooks, regardless of the textbooks of other subjects. A few people have concerned with the features of English textbooks in China where English is learned as a foreign language. In order to improve students’ multiliteracies in the process of gaining new knowledge in English class, it is essential to probe in the relationship between images and texts in English textbooks from the perspective of multimodal discourse analysis.Under the framework of Kress & Van Leeuwen’s Visual Grammar and Royce’s Intesemiotic Complementarity, this study conducts a multimodal discourse analysis of a set of textbooks-- Experiencing English Viewing, Listening and Speaking. This set of textbooks puts “viewing” in the first place with the purpose of adopting multimodal inputs to develop students’ multiliteracies, and highlights the role of image which takes a quite large proportion in the whole and is the most prominent difference of the English textbooks used in universities in China. Both a quantitative and qualitative study will be carried out to investigate the multimodal discourse in the four books. Firstly, a quantitative approach will be employed to deal with the statistics of all background colors and images, and the main multimodal content features of these textbooks will be analyzed. Secondly, the images will be interpreted and discussed in a quantitative and qualitative approach to examine how they construct the three meanings of Visual Grammar respectively. Last but not the least, the semantic relationship in meaning-construction between language and the image is analyzed under the framework of Intersemiotic Complementarity.The major findings of the present study indicates that, images in this set of textbooks reach the purpose of realizing Visual Grammar through construing the representational, interactive and impersonal meanings. The narrative representation is employed as the dominant type of representational meaning, as it describes what and how activities or things happen. Reactional process is more preferred among the three processes of narrative representation. In realizing interactive meaning, offer type overwhelmingly dominants the feature of contact. Intimate distance takes the most percentage in social distance. Both offer and intimate distance are helpful in establishing friendly relationship through involving the interactive participants together. The compositional devices include information value, framing and salience. The outstanding features of information value are suggested by given and ideal. With reference to salience, most images locate in the center position. Images in this set of textbooks are the information resource while the texts provide some complementary information for listening, so they are more salient than texts. The framing device--clear lines is the most frequently used, as it makes the layout clear and tidy.According to the results of relationships between images and texts under the framework of Ideational Intersemiotic Complementarity, repetition and synonymy are the main types of semantic relations. Repetition is the most frequent interrelation and synonymy follows as the second. Through analyzing these two semantic relations, these interplays between images and texts function as explanation and complementation for each other in meaning-making.Several implications for improving students’ multiliteracies are provided based on the results of this study. Textbooks provide students with knowledge and academic skills, and they can be used to cultivate students’ multiliteracy as well. Through the multimodal approach of learning, students can get a deep understanding of the new knowledge and practice the skill of knowing multimodality as well. Based on students’ cognitive competence, teachers need to analyze the textbooks deeply to understand and make the best of these multimodal teaching approaches in the textbooks, and enhance students’ learning interests by strengthening the effect of classroom teaching. With a better understanding of the multimodal discourses in textbooks, teachers can accomplish the pedagogical goals more easily.