A Framework for the Rapid Creation of Quality-Assured Programming Rubrics for New K-12 Computer Science Teacher

When this research began, AP Computer Science comprised only 0.9% of all AP tests taken in 2014, with roughly 39,000 students [Col97]. After initiatives such as America Competes, CS10K, CSforAll, and the official launch of the new AP CS Principles course in 2016, this number jumped to 2.1% or 104,000 students. This tremendous surge in Computer Science enrollment is a success for the programs, but also reflects the rapid rise in the number of K-12 teachers teaching computer science.;Because of the increasing demand for new CS Principles teachers, they have been recruited from diverse backgrounds.Many CS Principles teachers do not have any background in Computer Science and others have taken just one or two programming courses in college. This lack of experience makes it difficult for new teachers to identify learning goals and provide student feedback on programming lab assignments.With the rapid growth in novice Computer Science Principles teachers, new resources are needed to help teachers not only identify the computational thinking learning objectives in student lab assignments, but also to help teachers grade these programming assignments.;For CS Principles (CSP), several curricula were first designed for the college level and then used as a basis for high school CSP classes. This means that lab assignments were not already annotated with CSP-aligned learning objectives. I sought to provide rubrics that would help new teachers grade the CSP labs and give students feedback on whether they were achieving the CSP learning objectives. I first systematically made rubrics based on common Computer Science grading standards for auto graders and intelligent tutors used in college programming courses. Through this process, I found that the standards used for college Computer Science courses were not detailed enough to assess beginner-level computational thinking in projects. The small portion of the initial rubric focused on learning was biased towards expert programmers as opposed to those still learning the beginning skills taught in CS Principles. Therefore, I reoriented the rubric-making process to be focused on targeting CSP learning objectives.;In order to determine the learning objectives associated with each lab assignment, I applied a Delphi method to poll experts through a controlled group decision making process. Delphi participants generated both associated learning objectives and expected code samples for CSP labs. These were then grouped into topical categories (abstraction, conditional logic, etc.) to develop learning-oriented rubrics. When master CSP teachers and CS undergraduates (novices) used the Delphi-created rubric to grade the Hangman and BrickWall labs, they achieved a high level of interrater reliability. Although the Delphi-created rubrics resulted in consistent grading, the process.;Through a series of reliability studies on the initial rubrics, I found that trained Computer Science undergraduateswere as reliable as the CS PrinciplesMaster Teachers using the rubrics, and could act as surrogate 'almost experts' in systematically generating 32 learning-based rubrics using the new methods. I tested these new learning-based rubrics with active CS Principles teachers. I provided teachers high, medium, and low samples of student work, and had teachers mark in code where they were looking for the associated computational thinking concepts. I analyzed the consistency of grade distributions between graders, and also created a visualization to investigate the reliability and usefulness of the rubrics. The visualization helped me verify that teachers using our learning-based rubrics were on the right track for identifying computational thinking in code, but it also revealed that more support is needed for novice CSP teachers to grade code samples that were substantially different from the typical correct solution. These results confirmed that the modified Nominal Group Technique using almost-experts is sufficient to create learning-based rubrics that are reliable for assessing computational thinking in code.;I next applied Baartman's Wheel of Competency Assessment (WoCA) to further investigate the validity and appropriateness of the learning-based rubrics.UsingWoCA, the rubrics can be measured on 12 different quality criteria, including fitness of purpose, cost-effectiveness, meaningfulness, and cognitive complexity.Most of the criteria (10.5 of 12) focus on usability and appropriateness for teachers and the course content. The newly developed learning-based rubrics meet all 10.5 of these teacher-focusedWoCA criteria. This analysis shows that the newly created learning-based rubrics are a validated method of support for identifying and understanding learning objectives in student code by novice CS Principles teachers. (Abstract shortened by ProQuest.).