Critiquing as a cognitive task analysis methodology

This research investigated critiquing of a novice by an expert as a knowledge elicitation (KE) technique for cognitive task analysis (CTA). This technique takes advantage of the ability to critically assess another's task performance, which is often conducted by more skilled practitioners to train less skilled practitioners.; CTA methodologies are used to discover expertise that domain practitioners utilize to perform tasks, but are unable to reliably articulate when asked directly. Specifically, CTA's identify ineffective strategies that lead to poor performance (i.e., a model of mistakes that “novices” make), as well as adaptive strategies that have been developed by highly skilled practitioners to cope with task demands (i.e., a model of “expert” skill).{09}KE, a subset of CTA, uncovers information about how a practitioner works in a field directly from the practitioner.{09}However, KE has the challenges of grounding in context, accessibility to experts and relevant tasks, being laborious and time consuming, and repeatability. The critiquing methodology was investigated for its ability to address these challenges as well as for having any specific advantages.; In this study, six experienced intelligence analysts critiqued a junior analyst's process of determining the causes and consequences of the Ariane 501 rocket launch failure. Protocols were generated from each critiquing session. Each protocol included the script, a transcript of verbal critiques, and a transcript of opportunistic questions and responses. Analyses were conducted using process-tracing analysis of responses to the novice's comments and actions. The domain findings revealed that the domain practitioners experience challenges with problem definition, need to develop mature search methods, rely on collaborating with other analysts and must have systematic, traceable methods of verifying source credibility and hypotheses.; The critiquing method proved to be authentic, as the gathered information was indicative of the domain, and generative, as the results pointed to potential methods to support the domain practitioners. The findings indicated the critiquing method addresses KE shortfalls and has specific benefits. The study brought up the interesting concept of ‘community of expertise’ that supports sharing revealed knowledge about one domain with other domains. Specific considerations for using the critiquing methodology are discussed.