Evaluating DETECT indices and item classification using simulated and real data that display both simple and complex structure

Dimensionality assessments are often conducted to validate a construct, which also has implications for diagnostic testing (e.g., Tate, 2002). DETECT is a nonparametric dimensionality assessment procedure with two indices, Dmax and rmax. The indices are used to assess the strength of multidimensionality and whether the dimensional structure identified is simple or complex. DETECT has been shown to work well with test data of simple or approximate simple structure (e.g., Zhang & Stout, 1999b). However, its performance with data of complex structure has only been evaluated in one published study (Gierl, Leighton, & Tan, in press). The present study evaluated the performance of DETECT under conditions of approximate simple and complex structures using simulated and real data. The impact of three factors on the performance of DETECT was investigated---degree of complexity in data structure, correlation between dimensions, and sample size.;In the simulation study, a 3 x 4 x 3 fully crossed design was used. The effect of the three factors on Dmax, rmax, classification accuracy and classification consistency, were studied. Regression analyses for both Dmax, and rmax, regressing on classification accuracy, were used to find new critical values for Dmax and rmax. In the real data study, DETECT was used to analyze the SAT 2005 March administration data with hypothesized dimensional structure to confirm results found in the simulation study.;Results from the simulation study suggested that DETECT could adequately identify the dimensional structure of tests (with 80% or higher classification accuracy and consistency) for 15 of 24 cases under the approximate simple structure conditions and 10 of 48 cases under the complex structure conditions. While sample size did not have a significant effect on DETECT results, the other factors all affected DETECT results significantly. Relaxed evaluation criteria of 0.15 for Dmax. and 0.60 for rmax were proposed based on results from the regression analyses. Results from the real data study agreed with the simulation results, and thus indicated the simulated conditions were realistic. Implications to researchers and practitioners were given based on the simulation results. Limitations of the present study and future directions were also discussed.