Technical Performance Measurement and Technical Risk Analysis Utilizing the Assessment of Probability Distributions From Expert Judgment

Systems engineers are routinely tasked with facilitating the delicate balance between cost, schedule, and technical performance in the acquisition of new technology. While techniques for assessing cost and schedule performance abound, the estimation and risk analysis of technical performance in development programs currently rely on the unstructured opinions of experts because the identification and application of relevant quantitative data for constructive modeling is not practical. This research proposes a methodology that combines existing procedures for technical performance measurement with Cooke's Classical Model, a method for mathematically aggregating structured expert judgment, and Garvey and Cho's TPM Risk Index, a method that normalizes technical performance measures referenced to its desired level of performance, to develop predictive progress plans for technical performance estimation and risk analysis. Cooke's Classical Model has been widely used in risk analysis for over 20 years; however, the model's validity has been the subject of much debate. This study conducts a comprehensive examination of the model through an iterative, cross-validation test to perform an out-of-sample comparison between the Classical Model and the equal-weight linear opinion pool; followed by a case study to demonstrate the proposed methodology using actual data from a Department of Defense acquisition program. The results indicate that Cooke's classical model significantly outperforms equally weighted expert judgment and that the proposed methodology is suitable for developing predictive technical performance and risk progress plans for systems engineers.