Psychometric properties of aviation safety performance evaluation instruments: Dependability of assessments

This dissertation presents research on the psychometric properties of the instruments used by Federal Aviation Administration (FAA) safety inspectors to assess the cockpit performance of airline crews. While psychometric studies have been conducted in training programs administered by the airlines themselves, little research has been done in this area on the processes and tools used by the FAA. The limited research that has been conducted suggested that the interrater reliability (IRR) of current assessments could be questionable. A version of the instrument currently in use by the FAA and two other instruments were used by active FAA inspectors who viewed eight videotaped scenarios of staged cockpit activities. Generalizability Theory (GT) analysis (Cronbach, Gleser, Nanda, & Rajaratnam, 1972) was chosen because of its ability to address multiple error sources in a single analytical framework. The main effect for raters was less than expected, normally a sign of good MR, but a substantial interaction effect was found between raters and the rating situation. An expected substantial effect was found between types of instruments, but the individual performance of each was found to be nearly equivalent in terms of allocation of error sources and generalizability coefficients. Further substantial situation-dependent effects were found when analyses were conducted for individual scenarios. The findings that the assessments made by these raters were highly variable between situations and types of instruments suggest problems with construct validity and definition of the dimensions of performance being evaluated. While there appears to be some consistency in the patterns of overall scoring, there appears to be considerable ambiguity with regard to measured constructs and related operational definitions that underlie the instruments' scales.