A COMPARISON OF BASIC SCIENCE ITEMS AND CLINICALLY RELEVANT ITEMS IN MEASURING PHYSICIAN COMPETENCE

This study investigated how test items which deal with basic science material perform when included in a national certification examination being administered to candidates seeking specialty certification by the American Board of Emergency Medicine.;Eighty basic science items were distributed throughout three booklets of four alternative, single best answer multiple choice test items. These items were for experimental purposes only and did not affect the candidate's pass/fail outcome.;A total of 616 candidates sat for the examination. Candidates qualified for certification by completing five continuous years of emergency medicine practice or by successfully completing an approved emergency medicine residency program.;Four primary test item scales were of interest. The Basic Science scale, a scale of high clinical relevance (HCR) items which had been determined to correlate highly with performance on oral stimulated patient encounters (the criterion), low clinical relevence (LCR) items which had no discernible statistical correlation with the criterion, and medium difficulty (MD) items which had mid-range levels of difficulty (p values). Any items that might have been redundant (e.g., MD and LCR) were not included in the analysis.;Although each of the four scales discriminated between the two subject groups, they did differ in relative discriminating power. The results showed that HCR items did not discriminate between levels of competency as well as was expected. Using discriminant analysis, it was found that the MD scale was superior in making discriminations. The BS scale was second best and when combined with the MD scale created the most efficient collection of items. The LCR and HCR scale made no significant contribution to the MD/BS composite.;Although the BS items proved to be slightly less correlated with the criterion than were the LCR items, they nonetheless proved to be superior to even the HCR items. It was also found that there were no differential effects of the four BS subscales (biochemistry, microbiology, pharmacology, and physiology).