Examining the construct validity of readability: Usefulness of readability information for the validation of multiple choice tests

The validity of readability indices for the measurement of the reading level of multiple choice test questions was examined. Three indices were applied to items from tests of job knowledge and reading ability. Tests were administered to 174 applicants for three maintenance craft positions at a large petrochemical refinery to determine actual test item difficulty.;It was hypothesized that the reading and cognitive ability level of the subjects used to generate item difficulties would moderate the relation between item reading level and item difficulty and that mechanical ability level would not moderate this relation. The comparison of sub-group correlations revealed no significant moderator effects. Regression analyses revealed that expert judgments of difficulty accounted for small but significant variance in job knowledge test item difficulty, while reading level did not. In some conditions, reading level accounted for significant variance in the difficulty of reading test items and judgments accounted for additional variance.;In general, reading level measures accounted for little variance in test item difficulty. Limitations and implications of the study are discussed and future directions are suggested for the measurement of the reading level of multiple choice test items.