A simulation study of the Classical Method of expert judgment combination: How many seeds and how many experts

This dissertation is an investigation of selected aspects of the Classical Method of expert judgment combination: a weighted linear average where the weights are determined by rating the experts on the basis of a test in which experts provide subjective probability distributions representing their best estimates of a parameter or physical quantity. These distributions provide information about how precisely the expert knows the value of the quantity as well as the level of certainty that the expert expresses. The Classical Method makes two measurements from the set of provided distributions, calibration (measuring correctness) and "information" (measuring distribution width). Two parameters of the Classical model are investigated: number of experts and number of test questions. The data show that there is a benefit in adding experts up to ten which is in accordance with theoretical analysis of linear opinion pooling. The data also show an increasing benefit in using up to fifteen test questions, with a continuing increase, but leveling off after fifteen.; This dissertation compares and contrasts the Classical Method calibration measurement with another alternative calibration measurement from the literature (Hora), both theoretically and empirically via a simulation study. There are significant differences between the calibration methods which have an impact on the final combined CDF. This dissertation investigates the usage of both calibration methods to determine the expert weights and as a method of evaluating the resultant combined CDFs.; A simulation algorithm for randomly generating probability interval data of the type used in an expert judgment study of a continuous quantity is developed (GENIUS). Expert provided distributions are not assumed to conform to a specific probability distribution; rather, the simulation models cognitive and mental properties of experts known from the literature: normative (statistical methodology) expertise and subject matter expertise; expert overconfidence; expert bias; and a tendency towards symmetry in responses. The simulation method presented can be used in conjunction with any mathematical expert combination technique operating on three point interval data and allows user parameter adjustment via spreadsheet. Full code used to perform the simulation and data analysis is also presented.