Development And Validation Of Note-Taking And Outline-Summary As A Classroom Test To Measure Listening Comprehension

The thesis develops and validates a classroom listening test consisting of two tasks of taking notes while listening, and making outline-summary after listening. The new Note-Taking and Outline-Summary test is intended to measure, more validly and reliably, student's listening comprehension of lectures, and to improve the beneficial washback of classroom listening testing upon teaching and learning of listening.The thesis consists of six chapters. The first three introduce the process of developing the new test, while the latter three discuss the results of its validation.Chapter One introduces the research background by pointing out the potential harmful impact of misusing unfitting MC test and Note-Taking & Gap-Filling test to measure academic listening. Then, four research questions are proposed: (1) How to guarantee the authenticity of listening test tasks? (2) How to solve the dilemma between sampling quality and rating reliability? (3) How to gather and interpret sources of validation evidences, so as to justify or nullify the test's intended construct? (4) Why are MC and NT&GF inappropriate measures of academic listening?Chapter Two provides a systematic theoretical framework for test development and score interpretation. Working definitions for test development are discussed via a three-step framework by Bachman (1990): (1) Defining theoretical construct; (2) Specifying testing objective; (3) Establishing scoring criterion and choosing scoring method. The score interpretation follows Messick (1980) in regarding construct validity as the unifying concept for all the indicators of test quality.Chapter Three elaborates upon the development of the specifications for the new test: (1) The theoretical construct is candidate's ability to understand the macro-structure and illustrations of the lecture. (2) The specific testing objective is to require candidates to make an outline-summary of the lecture on the skeletal outline sheet, after they listen to the lecture and take notes. At the stage of test designing, discourse analysis is made of all the lectures, to ensure that the structure of outline sheet is highly consistent with the lecture's discourse structure. (3) The scoring criterion is established as to score candidate's performance on the two different levels of macro-structure and illustration. "Acceptable Word Method" (Shu 1999) is used to score each item on the skeletal outline. The score on either level is the percentage of candidate's correct responses in proportion to all the items on this level. When necessary, an overall score can be generated by averaging the candidate's score on the macrostructure level and score on the illustration level.Chapter Four introduces the design of seven pairs of fourteen experiments in terms of preparation, administration, data collection and statistical analysis. By introducing subjects, materials, instruments, and detailed planning and procedure of experiments, this chapter paves the way for the discussion of experiment results.Chapter Five first analyzes the results of pretests which confirm that all the four classes are identical to each other in general listening proficiency. The chapter interprets validation evidences gathered from four sources of test content, test internal structure, response type, and relationship to external criterion, so as to investigate whether the new test enjoys task authenticity and to discuss what measures should be taken to balance between sampling quality and rating reliability. Besides, the chapter discusses validation evidences obtained from administering MC and NT&GF tests, in an effort to probe for sources of invalidity and reasons why these two tests are inadequate measures of academic listening.On the basis of discussion of validation evidences, Chapter Six gives answers to the four research questions: (1) Note-taking is authentic in that its function of relieving cognitive burden during listening was proven with a pair of experiments, besides it is a common activity accompanying academic listening. As for the skeletal outline sheet with minimal written prompts, performance on it was proven to be not significantly different from performance on blank sheet by the control group who made their own outline-summary, this result attests to the task authenticity of making summary on skeletal outline sheet. (2) It is exactly the layered, appropriate-word, discrete-point objective scoring of skeletal outline summary that solves the dilemma between sampling quality and rating reliability. Two carefully designed parallel forms were administered, and parallel-forms reliability (which is also the criterion-related validity between the two tests) was calculated to verify that the hierarchically arranged format of skeletal outline sheet ensures the representativeness and coverage of sampling of candidate responses, and its layered objective scoring method guarantees that high rating reliability can be achieved during scoring of the test. (3) The argumentation process of this thesis exemplifies how to gather and interpret various validation evidences by different approaches such as theoretical rationalization, empirical experiments and discussion of their results, test content analysis, and simulation of test-taking process. Analysis and interpretation of various validation evidences contribute to the justification of the new test's claim of measuring the intended construct of academic listening comprehension. (4) Two reasons account for why MC and NT&GF are inappropriate measures of academic listening: inauthentic or under-representative sampling of responses and too many written clues that introduce construct-irrelevant variances to the test. These two factors may incur test-wiseness and contaminate the validity of using these two tests.The thesis concludes with an appeal for mutually beneficial interaction between classroom listening testing and the teaching and learning of listening comprehension.