Text difficulty of rating scales used by school psychologists

Rating scales are often among the most frequently administered assessment instruments across various settings including schools, clinics, and forensic settings. Review of the literature suggested limited research has been conducted to examine the text difficulty of the directions and the items of rating scales used in schools. The purpose of this study was to examine the text difficulty of directions and items of 295 rating scales (86 parent-report rating scales, 82 teacher-report rating scales, 94 child and adolescent self-report rating scales, and 33 parent-/teacher-report rating scales) using both traditional measures of text difficulty (e.g., readability formulas) and modern measures of text difficulty (e.g., word characteristics or cohesion metrics). The text difficulty of rating scale directions and items were analyzed using Coh-Metrix (Grasser, McNamara, Louwerse, & Zhiqiang, 2004), a computational text analysis program. Directions, consolidated items (i.e., all items of a rating scale), and individual items of each rating scale were analyzed independently. Coh-Metrix yielded 2 traditional text difficulty metrics and 5 modern text difficulty metrics to analyze directions and items. In addition, 4 modern measures of text difficulty were used to analyze the directions only. A MANOVA revealed significant effects for the type (i.e., parent-report, teacher-report, child and adolescent self-report, or parent-/teacher-report) of rating scales, level (i.e., directions and items), and the interaction between type and level of rating scale. Post hoc discriminant analyses suggest that child and adolescent self-report rating scales generally had the lowest text difficulty (across consolidated items and directions and items only) of all four types of rating scales, with Flesch Reading Ease and Word Frequency providing the best discrimination from other types of rating scales. Teacher-report ratings scales generally had the highest text difficulty (across consolidated items and directions and items only) of all four types of rating scales, with Age of Acquisition, Word Concreteness, and Word Imaginability providing the best discrimination between teacher-report and parent-report rating scales. Across rating scale types, the text difficulty of directions was generally higher than the text difficulty of items. Implications for these findings and directions for future research are discussed.