An assessment of using response latency to estimate levels of certainty associated with a production rule in expert systems

Knowledge elicitation is a bottleneck in the expert system building procedure as it is expensive and time consuming. It involves interviewing a human expert over a period of time to discover decision rules as well as the relative measures of certainty or preferences among the expert's recommendations. Certainty Factors (CF) are associated with these rules and are used to measure the expert's degree of certainty or preference.; The research problem is directed at assessing two knowledge elicitation methods that can be used to measure an expert's CFs and the degree of inconsistency in a set of CFs, called the Certainty Ratio (CR). One method is the older or classic method where the expert self-reports the level of certainty associated with a rule. The other one is based on using response latencies (the time taken for an expert to express a preference in one recommendation over another). The Response Latency method is newer and has the advantage of being unobtrusive, less prone to conscious censure, quicker to perform, requires less effort, and possibly is less expensive to administer. If the response latency method performs at least as well as the classic, self-report method, then it can be used in place of the classic method.; To determine if there is a difference in the two methods, 21 subjects were evaluated in a paired sample design. Employing a computer-administered questionnaire, subjects expressed relative levels of confidence for a set of potential recommendations and the time for each subject to respond was recorded unobtrusively.; Using paired sample t-tests, sufficient evidence was found to reject the null hypotheses (α =.025) that the mean of the difference of the CFs was zero in the population (p <.0001). Additionally, there was sufficient evidence to reject the null hypothesis (α =.05) that the mean of the difference in the CRs was zero in the population ( p <.0001), accepting the alternative hypothesis that, in the population, response latency has a lower mean CR than the self-report method. Thus, there is initial support for using the response latency method in place of the self-report method of knowledge elicitation.