A New Perspective On Detection Performance Decline

Aberrant responses,which are caused by respondents’ unusual behaviors(e.g.,carelessness,speededness,item pre-knowledge,warm-up,copying answers from neighbors,performance decline),are frequently observed in various testing programs.On the one hand,aberrant responses will lead to improper assessment of the latent trait of respondents,on the other hand,it will also affect the accuracy of item parameter estimates and test validity,and even result in poor overall fit and item-model fit.Thus proper modeling and detection of aberrant responses is crucial both theoretically and practically in psychological and educational measurement.Previous works have primarily approached the aberrant responses research in terms of developing person fit statistics and proposing models to capture the aberrant behaviors.This dissertation attempts to incorporate change point analysis(CPA)to detect one type of aberrant responses:performance decline(PD).PD refers to that respondents fail to try their utmost effort to attempt each item due to personal factors(e.g.,gradually decreased motivation)and environmental factors(e.g.,test time constraints),resulting in poor performance at the end of a test.De La Torre and Deng(2007)found that PD detection fail to achieve acceptable power,and the highest power is 0.55.Additionally,existing three CPA statistics cannot be directly used to detect PD,because they are appropriate for two-sided alternative hypotheses.To address these issues,a new CPA statistics to detect PD,which is named as J Smax,was developed by analyzing the principle of CPA statistics to detect aberrant responses and enlightening from the ideas of the Bayesians.Then,existing three two-sided CPA statistics were converted into one-sided statistics to accommodate PD detection.To explore the performance of the proposed CPA statistic,this study use Monte Carlo simulation method to obtain the critical values of the new statistic under different test lengths,as well as the power and the type-I error rate under various experimental conditions.Then,compare it with one-sided CPA statistics.Results show that the new CPA statistic can detect PD with higher power while generating a well-controlled typeI error rate.Compared against three other one-sided CPA statistics,the new CPA statistic exhibits a comparable type-I error rate and a gain in power from 1.0%to 8.2%.Finally,this study uses two empirical datasets,Program for International Student Assessment and the Advanced Raven Progressive Matrices Test,to demonstrate its utility in the actual tests.Additionally,this research uses the graphic method to verify the correctness of the classification results of JSmax.Through the display of the graphic method,researchers can intuitively judge whether the respondent is PD.The results show that the detection results regarding PD by the four CPA statistics have high consistency,and J Smax can indeed detect the subjects experienced PD successfully.