A comparison of the Rasch model and the three-parameter logistic model applied to the quantitative subtest of the General Aptitude Test, Saudi Arabia

Today, item response theory has increasingly become the leading testing framework for developing large-scale standardized tests. The many advantages of item response theory have made this framework the best test model available. For instance, in addition to the major advantages and distinctions of IRT over CTT (e.g., invariance), item response theory when applied to test data can enable advances in test construction, test equating, identifying potentially biased test items, computer adaptive testing, and item banking. Unfortunately, these advantages of the item response theory cannot be obtained successfully unless adequate fit to the data of interest is achieved. This study attempted to compare the applicability of the Rasch and the three-parameter models to the quantitative subtest of the General Aptitude Test, Saudi Arabia. Seven research questions were proposed. All research questions were answered using multiple methods proposed by the Rasch and three-parameter model's proponents. Methods were divided into three phases. The first phase involved testing and evaluating the assumptions of the Rasch and three-parameter models. The second phase included testing and evaluating the invariance of parameters. The third phase included two parts: (a) evaluating the reliability of item and ability parameters and (b) evaluating the Rasch and three-parameter model predictions and examining the test information produced by the Rasch and three-parameter models.; The study found that there was some level of guessing and item discrimination did vary to a minor degree. The three-parameter model showed better overall fit to the data than the Rasch model and item-level residuals were smaller. Those results support use of the three-parameter model. On the other hand, invariance, item parameter accuracy, and item (and test) information results favored the Rasch model. And, the Rasch model seemed robust with respect to minor violations regarding guessing and item discrimination. Further, the calculation of model fit, which favored the three-parameter model, is contrary to how fit would be calculated under Rasch software and thus yields unclear evidence. Overall, since the invariance item parameter feature of IRT was violated with the three-parameter model, it was concluded that the Rasch model was more applicable to the small sample quantitative subtest data than the three-parameter model.