| Data from 3,266 flight observations aboard regularly scheduled domestic and international flights from five U.S. airlines in both standard and automated aircraft are presented. Trained raters assessed flight crew performance on a four point scale (1 = poor, 2 = minimum expectations, 3 = standard, 4 = outstanding) in the areas of leadership, crew interaction, and automation management. Raters de-identified all observations, and crews were under no jeopardy of retribution for any negative behavior they exhibited during the flight. Seven core measures of crew performance, and four core measures of automation management were extracted from regression analyses and a review of commercial airline accident and incident investigations. Examination of the data indicated that less than 14% of the flights observed were rated as below standard, 71% were rated as standard, and 15% were rated as outstanding. Substantial differences in crew performance were found as a function of airline. Crew performance was also found to vary as a function of the quality of flight briefings, the complexity of the operating environment, the degree of crew familiarity, and the length of the flight. Crews in automated and standard aircraft were not found to perform differently on core measures shared by both aircraft types. Large performance differences between airlines were found for core automation management measures, such as establishing guidelines for automation management and using automation appropriately. Additionally, higher scores on the automation management measures were associated with good flight briefings. Further analysis shouwed that there were pilot flying effects in complex operating environments: crews performed better when the first officer was the pilot flying and worse when the captain was the pilot flying. Principal Component Analysis was used to derive three higher order scales of crew performance: Command, Crew Interaction, and Automation Management. Cluster analysis indicated that there were three natural groupings of crews based on their performance. Crews in automated aircraft had High, Standard, and Low groupings, and crews in standard aircraft had Above Standard, Below Standard, and Low groupings. Further analysis indicated that the introduction of automation has posed new issues and modes of possible errors in crew performance, and that automation can have either very good or very bad effects on crew performance in modern aircraft. Recommendations for training and policy interventions are discussed. |
