The Empirical Study On Visual Design Of Verbal Information In CALL And Its Effect On Listening Comprehension

Most of the multimedia designs in CALL cannot get rid of the binary opposition between graphic and text. No matter how good the design is, it seems just to try to blend or eliminate the opposition. Understanding a language possesses indivisibility and multimodality. Studies on multimodal teaching have achieved a lot so far, but a majority of them are designed for scientific content rather than language teaching. When it comes to visual designs of text and the structure of language, there are fewer researches. According to the dual coding theory and cognitive theory of multimedia learning, the modality specific effect lies in the understanding of verbal information and the combination with superiority effect lies in multimodal representations. Multimedia designs should focus on visual designs of verbal information by using timing or node as a link to synchronize sounds with texts, movement, and situation, etc. instead of the richness of multimodal representations in CALL. The ecological regression of language communication brought by the technology might be able to restore the artificial environment of language acquisition, which may also be one of the important subjects of CALL. Thus, based on dual coding theory and cognitive theory of multimedia learning and, the article employs a combination of qualitative and quantitative approach to discuss the differences of various visual designs of verbal information in the effectiveness of listening comprehension. To solve the problem, the study mainly checks(1) Does the modality specific effect lie in the understanding of verbal information and why is that?(2) Which combination of multimodal representations has superiority effect?(3) What are the specific conditions and theoretical foundations to realize superiority effect of multimodal combination?The experiment has designed three kinds of multimodal combinations(narration + captions + videos, narration + videos and narration + captions) to test comparison among audiovisual materials representing three text types including the expository text type, argumentative text type and the instructional text type respectively. Based on the classification of visuals(Bejar, et al, 2000), context videos have been chosen in the experiment. The subjects participated in the present study are chosen from the freshmen of School of Foreign Studies Jiangnan University, Jiangsu Province. Altogether 116 students from six classes attend the reading and listening proficiency test, and the researcher selects 51 subjects with no differences in the scores of the proficiency test among which six students take part in the pilot study and 45 students in the formal experiment. The remaining 45 were randomly divided into three groups. The subjects watch three passages before taking the listening comprehension test. The paper version of the items of the listening comprehension test will be handed out to the test-takers before playing the audiovisual materials, and each watches the three segments of different materials in three different processing modes. Each group watches three different audiovisual materials which are played only once with three different treatments. The subjects are told to take a listening comprehension test composed of 18 multiple-choice questions before watching the materials. Each material is matched with six questions. The listening comprehension test is conducted in Multimedia Language Learning Lab. Each student uses one computer screen and one headset. Finally, six subjects were randomly selected to participate in the interviews about the listening comprehension test in which the interview questions cover the effect of three different representations on their listening comprehension and the difficulties they encountered during listening and reading.The following conclusions are drawn from the experimental data analyzed by One-Way ANOVA in SPSS. Modality specific effect does exist in the understanding of verbal information. Various representations will get different results on reading and listening while the learners are watching audiovisual materials through different visual designs: The combination of narration + videos has superiority effect and achieved the most satisfactory. The combination of narration + full captions is inclined to generate perception load and split attention effect with inferior listening effect. The downside of the audiovisual materials presented by narration + full captions is the substantial loss of verbal information in the absence of contextual clues and the combination is less likely to realize multimedia effect with the worst listening results. The data analysis reveals that when it comes to the instructional videos, the superiority effect of narration + videos manifests obviously while when it comes to the expository and argumentative materials, there is no significant difference on the comparison among three visual designs. The superiority effect of narration + videos has been much weakened when the combination is used to represent the expository text type. The possible reasons may be that most of the instructional materials are strong context with graphical representation, movements, and procedural demonstration, etc. while the expository and argumentative materials are more abstract with low context correlation between visual representation and the logic structure.The study confirms modality specific and the inductive coupling among various combinations. Essentially, verbal information has modality features on both audio and visual signals. The combination of narration + videos stimulates relative information through dual channels(visual images and audio narration) with simultaneous processing so that students will have the extra attention to process information on the screen instead of overloading in one channel. Images are presented simultaneously with relevant words to generate spatial and temporal contiguity of chronological signals, and the integration of words, images and prior knowledge will be able to produce the word and corresponding image representation at the same time in order to get a better listening results. As to the combination of narration + caption + videos, both captions and videos belong to the visual channel. Two signals(captions reading and images recognition) in one channel give rise to perception load and split attention. When learners do not have sufficient listening capability to understand, they will have to seek help from captions, leading to the omission of details of the images processing and impaired reading and listening, which affect the integration of meaning in the listening comprehension. The downside of the audiovisual materials presented by narration + captions is the substantial loss of verbal information in the absence of contextual clues. By now, multimedia effects abate with the increase of peripheral interferences. Captions become redundant information, resulting in poor reading and listening results. Students either listen with closed eyes or use speed reading without listening.The inductive coupling of multimodal learning confirmed in this study makes a difference to teaching practice of the sensitivity to different video types. There is a huge difference in the context correlation between visual representations of various video types and video contents. Not all video can be effectively used for the teaching of listening. The results of the study suggest that visual designs of verbal information shall choose the corresponding multimodal combination according to different text types in addition to other indicators such as the readability of reading and listening materials, the degree of difficulty, and the applicability in order to achieve the best training effect.