Formative Evaluation Of Foreign Language Programs: Establishing A Systematic Procedure And Exploring Its Effectiveness

As an important component of language programs, formative evaluation aims at improving language teaching and learning through providing teachers and learners with immediate and effective feedback. Studies of the theory and practice of formative evaluation of language programs have been carried out both at home and abroad. But compared with a large number of empirical studies conducted abroad, formative evaluation of language programs is rarely reported in China. In particular, the feasibility and effectiveness of formative evaluation in the context of foreign language teaching in China need further investigation. The present study focuses on the establishment of a systematic formative evaluation procedure for foreign language programs, and the investigation of whether and how the evaluation following the procedure promotes the development of the undergraduate program under evaluation. It is expected that the present study will contribute to the research on formative evaluation in general, and to the application of formative evaluation by foreign language teachers in China in particular. To this end, the study will address the following research questions:(1) What is the systematic procedure to be followed in the formative evaluation of a foreign language program in China? What are the main features of the procedure?(2) How effective is the proposed procedure when used for the formative evaluation of a foreign language program in terms of its impact on the curriculum design, classroom practices, materials of instruction, learning environment, etc.?(3) How effective is the proposed procedure when used for the formative evaluation of a foreign language program in terms of its impact on the teachers'understanding of the program requirements, teaching attitudes, teaching methodology, teaching content, etc.?(4) How effective is the proposed procedure when used for the formative evaluation of a foreign language program in terms of its impact on the students'understanding of the program requirements, learning attitudes, learning strategies, learning content, etc.?After a brief review of the educational literature on program evaluation, several models of educational program evaluation have been singled out for the exploration of a systematic procedure of formative evaluation in the present study, including Cronbach's three functions of evaluation (1963, 1983, 2000), Stufflebeam's Context-Input-Process-Product model (1971, 1997, 2000b), Scriven's goal-free evaluation model (1972, 1994), Stake's responsive model (1975), Wolf's judicial model (1975, 1983), Eisner'connoisseurship model (1975, 1983) and Weiss'Interest- Ideology-Information analysis (1983, 1997). On the basis of an in-depth analysis of the above evaluation models, the researcher develops a preliminary formative evaluation procedure for foreign language programs, which is to be validated and modified in the present study. In the study, the researcher follows the procedure in the process of implementing a one-year formative evaluation of a new undergraduate program for English majors at an institute in Dalian.The evaluation has been conducted in two stages over the course of two terms. By means of questionnaires, interviews, classroom observation, journals and proficiency tests, the evaluator has collected various types of data from the program designers, teachers and students in the treatment and control classes. The data have been analyzed and compared in great detail to show the impact of the evaluation on the program and on the EFL teaching and learning. To complement the qualitative analysis of the large amount of data collected, the questionnaire data and test data have been subjected to statistical analysis, including independent samples t-test, paired samples t-test and repeated measures ANOVA (analysis of variance). Following is a summary of the major findings of the study.The program designers have come to realize that formative evaluation is highly significant for revising and improving the program. For example, some changes have been made to the curriculum design, teaching arrangements, choices of teaching materials, etc. The undergraduate program has been reoriented towards the use of English for various practical purposes instead of English with an orientation towards computer technology. One new course aimed at improving students'English language skills has been offered in the English Department once a week. Profile test scores of the students have been made known to the teachers after each test so that they can help students identify their strengths and weaknesses. The teachers have also been required to meet regularly to exchange teaching experience and methods. Moreover, formative evaluation on a regular basis has been planned, which is expected to continuously benefit the language program and the EFL teaching and learning of the department.Improvements of teaching as a result of the evaluation have been made in the following aspects. First of all, computer-based teaching has been promoted in classes of intensive and extensive reading and writing. Secondly, the data from the journals written by the teacher participants indicate that more students have been encouraged to take part in classroom activities on their own initiative. Thirdly, the teaching content is not confined to the textbooks. The students are required to do supplementary reading, take notes and make comments on the reading materials. Lastly, the teachers are found to be more actively involved in research on teaching methods and self-evaluation methods.Progress made by the students as a result of the evaluation has been identified in the study. Firstly, their interest in learning English has grown and their motivation for learning has been increased. Secondly, they have learned to self-monitor their learning processes. Thirdly, they have a better awareness and understanding of autonomous learning and tried to develop more effective learning strategies. Fourthly, their sense of cooperation and teamwork has been improved. Moreover, the data seem to suggest that the impact of formative evaluation on EFL learning to a great extent depends on the purposes of the evaluation. With respect to the factors in the affective domain such as students'interest, motivation, attitudes, etc, formative evaluation has an indirect impact which affects the participants in a gradual way. However, when a particular arrangement or action of teaching or learning is evaluated, the effect will be more direct and prompt, and especially strong at the early stage of the evaluation.Based on the empirical study, the researcher modifies the preliminary formative evaluation procedure in several ways, for example, giving more emphasis to the analysis of the evaluation context, further specifying the evaluation steps, providing a more detailed description of the participants and instruments involved at each step, etc. The refined procedure has been defined as a synthesized, systematic and applicable procedure for the formative evaluation of foreign language programs.The procedure is a synthesized one because important implications have been drawn from the review of previous evaluation models. As a result, emphases have been given in the procedure to the analysis of the evaluation context, the careful division of the evaluation steps, the use of multi-methods for data collection, the involvement of participants, the roles of the evaluator, etc. The procedure is also a systematic one in that evaluators can design and implement formative evaluation of a language program in a phase-by-phase and step-by-step manner using different instruments on different participants for collecting different types of data. Moreover, the empirical data collected in the present study demonstrate to a large extent the effectiveness of the procedure, and it is expected that the procedure can be applied to the evaluation of language programs of a similar nature. As a result, some principal factors which may contribute to the success of program evaluation have been identified, including mainly the full support and understanding of the program director (s), the proper design of the evaluation which meets the specific requirements of the program, the active participation of all participants, and the evaluator's professional knowledge and skill of conducting the formative evaluation of foreign language programs.To sum up, the most important contribution of the present study is the proposal and validation of a systematic formative evaluation procedure for foreign language programs. The establishment of the procedure is of special practical value to the implementation of formative evaluation of foreign language programs in China, which are entering a new stage of reform in response to the changing needs of society. At present, few foreign language programs have incorporated formative evaluation as an indispensable component. The study not only raises the awareness of the importance of such kind of evaluation on the part of program designers and teachers but also equips them with a useful and practical framework. In actual implementation, however, modifications and adjustments to the evaluation instruments and methods are necessary to cater to the specific context of the program to be evaluated.