Scorers must qualify individually before they begin scoring any extended constructed-response items and for selected short constructed-response items depending on the level of complexity of the item and scoring guide. For such items, two sets of 10 to 12 papers are prepared that cover
The papers are randomly ordered within each set and identified with different numbering schemes between the first and second sets.
To qualify, scorers are required to achieve a certain percentage of exact agreement with the scores that were assigned to all responses in the qualifying sets. The general qualifying criteria applied for specific score-point levels are
Upon completing the qualifying sets and confirming that scorers have been deemed qualified to score, scoring begins. If a scorer does not qualify, the trainer can (a) allow the scorer to score on a probationary status, with articulated parameters (e.g., agreement, backreading and/or monitoring), (b) provide opportunities for additional training, or (c) request that the scoring content lead remove the scorer from the team for this item or for the project.