Developing A Model Of Cognitive Diagnostic Assessment For College EFL Listening

Cognitive diagnostic assessment (CDA) has great potentials in large scale testing. Withit, researchers in educational measurement and cognitive psychology are finally in a positionto design tests targeted specifically for providing valuable information about students’cognitive strengths and weaknesses (Leighton&Gierl,2007b).CDA had not been adopted in second language assessment until in recent several years,but very few cases in EFL listening research have been found. The dissertation makes a boldattempt to employ Cognitive Diagnostic Approaches (CDAs) to develop a listeningdiagnostic test model for fine-grained assessment and classification of the learner‘sknowledge state and presence or absence of certain attributes. The study also touches on thediagnostic report and the corresponding remedial guidance. The addressed questions are asfollows:1. What attributes count in EFL listening and how are they related hierarchically?2. How can we construct and validate the EFL listening CDA model?3. What are the considerations in presenting the online diagnostic reports?4. How can we guide the learners based on their diagnostic reports?The methodology falls into two phases. Phase I is analyzing the preliminary diagnostictest model (PDT). This includes attribute identification, Q-matrix construction, IRT andCDA analysis. In Phase II, the hypothetic model (H) is proposed based on Phase I and it isthen validated by comparing its3versions of hypothetic models, i.e. the test developer‘smodel (H1), students‘Think Aloud Protocol (TAP) model (H2) and7domain experts‘judgment model (H3).534test responses and the above three versions of hypothetic modelsare psychometrically analyzed using G-DINA model to find out their absolute and relativemodel-data fit statistics. The best-fit sub-model proves to be H3. After these exploratoryprocedures, the best-fit model is further verified with new data and its goodness of fit isconfirmed.Considerations for diagnostic online reports are also proposed, followed by four kindsof guidance for learners to cope with their diagnosed problems.The findings come as follows:To answer the first question, seven attributes were identified: phonological level (sounddiscrimination, stress and intonation); lexical level (less frequent vocabulary and oralexpressions); syntactic level (less frequent vocabulary and oral expressions); facts anddetails; main ideas; contextual and cultural related inference; note-taking and working memory. They are mainly independent attributes.How to develop the CDA model? It generally follows the4-step procedure of―attributeidentification‖â†'―Q-matrix construction and test item writing‖â†'―Psychometric analysisand verification‖â†'―Diagnostic reporting‖. However，the research demonstrated that―remedial guidance‖is a very appropriate and desirable destination for a diagnostic model.They five constitute a perfect system to empower learners in pursuing autonomy in EFLlearning.About the online diagnostic report,5considerations were proposed:1) offeringorientation information to the diagnostic score users;2) integrating the total scores into thediagnostic report;3) making diagnostic report files and scores interpretable, such as in theform of graphs, tables or other interactive ways;4) offering multi-dimensional analysis;5)including narrative interpretations for the unconventional diagnostic scores.Guidance is often given according to learner problems and diagnostic reports. It can bequestion-answer form of listening strategies, cases of successful learners, learners‘TAPreports and interview aural and visual of some world linguists.In the context of language learning and instruction, the availability of such diagnosticfeedback and guidance at a finer grain size would allow the instructor to identify thelearner‘s specific deficiencies and to plan instruction that is tailored to the needs of theparticular learner. It also facilitates learners on their journey to autonomy. In thelearner-centered and technology-assisted age of education, this study is especiallysignificant in its great potentials.