Assessing the impact of minimally intelligent, computer-generated, immediate feedback on an ill-defined problem solving task

Computerized learning environments successfully support problem solving in many well defined domains. Many such environments provide intelligent, immediate feedback whenever problem solvers commit errors that impede their progress toward a well-defined goal. The present research seeks to apply immediate feedback principles to ill-defined problem solving with Belvedere, a graphical environment to support scientific inquiry. Belvedere's normally on-demand help facility (the Coach) was modified to present immediate, intrusive feedback on basic inquiry principles while users created diagrams showing evidential relations among hypotheses and data. I predicted positive effects of immediate coaching on performance measures such as diagram completeness, diagramming errors, and information search measures, but negative effects on student attitudes toward Belvedere and its Coach. Undergraduate students in Experiment 1 used Belvedere with either intrusive coaching or no coaching. Performance measures showed mainly nonsignificant trends, most of which were in the predicted direction. Attitude measures showed more significant predicted differences. Latency measures suggested that students may not have attended to all of the immediate coaching feedback. Students in Experiment 2 used Belvedere with either intrusive coaching or standard on-demand coaching plus reminder prompts. Results were similar to Experiment 1 on both performance and attitude measures, with more marked group differences on the latter. Latency analyses suggested that students spent significantly more time processing requested coaching than intrusive coaching. Limitations of the Coach and of the problem solving task may explain the relatively few significant differences in performance measures. Results suggest that the affective costs of immediate feedback may outweigh its performance benefits for ill-defined problem solving, although this suggestion may be specific to the experimental task or to the relatively unsophisticated feedback currently provided by Belvedere's Coach.