EXPLORING REGRESSION SENSITIVITY IN A LABORATORY SETTING: SYNTHETIC DATA EXPERIMENTS AND ANALYSIS OF REGRESSION MODEL MIS-SPECIFICATION BASED ON A DYNAMIC SIMULATION MODEL OF ORGANIZATIONAL IMPACTS OF OFFICE AUTOMATION IMPLEMENTATION (POLICY ANALYSIS

Political discussions of public finding are increasingly based upon evaluations of program effectiveness which rely heavily upon elaborate statistical analyses. A disturbing number of regression-based program impact analyses show null results where program case studies show clearly positive results.;Recently it has been suggested that the above-mentioned null results may to some extent be due to strictly methodological artifacts rather than to aspects of the programs under study. To explore this, a mode of experimentation has been developed whereby synthetic data sets are generated by deterministic simulation models, the causal structure of which are fully known. Regression analyses of these synthetic data sets should predict the known causal structure in the absence of artifactual problems or assumption violations.;Early synthetic data experiments indicated considerable sensitivity of regression analysis to data quality. The present series of experiments focuses upon issues in regression specification. To this end, a simulation model of an organizational innovation (i.e. office automation in the U.S. Job Service) was developed and data generated for all key model variables. Targets for regression estimation were developed based upon the known structure of data generation. Regression models were specified, analyses run and comparisons made between expected and actual results.;Specification issues explored involved estimation of distal versus proximal predictors, nature of functional form and temporal specifications. The ensuing discussion of experimental results described masking effects of proximal predictors and mis-specifications of functional form for equilibrium and transient systems.;A number of suggestions are made for research beyond these preliminary regression specification experiments. And implications for econometrics and program evaluation are discussed.