| Two prospective randomized controlled studies of scientific curators of model organism databases (MODs) were conducted using common document collections to investigate the origins, nature, and extent of variation in curators' Gene Ontology (GO) annotations. Additional contextual data about curators' backgrounds, experience, personal annotation behaviors, and work practices were also collected to provide additional means of explaining variation. A corpus of nearly 4,000 new GO annotations covering 5 organisms was generated by 31 curators and analyzed at the paper, instance, and GO element levels. Variation was observed by organism expertise, by group assignment, and between individual and consensus annotations. Years of GO curation experience was found to not be a predictor of annotation instance quantities. Five facets of GO annotation quality (Consistency, Specificity, Completeness, Validity, and Reliability) were evaluated for utility, and showed promise for use in training novice curators. Pairwise matching and comparison of instances was found to be difficult and atypical, limiting the usefulness of the quality measures. Content analysis was performed on more than 600 pages of curators' hand-annotated paper journal articles used in GO annotation, yielding six types of common notations. |
