NAEP Technical DocumentationScoring Monitoring

			Within-Year Interrater Agreement Trend Scoring Backreading Calibration Archiving Scoring Materials

Careful monitoring of scoring is a hallmark of NAEP. Monitoring is conducted by a variety of qualified staff including:

• scoring trainers and supervisors at the team level,
• test development staff,
• psychometricians, and
• program management staff.

The type of monitoring practiced varies depending on the items being scored. For instance, items that were administered in a previous assessment are monitored for agreement across years (referred to as trend scoring) as well as within-year interrater agreement.

Ongoing monitoring of scorers is crucial to the NAEP goal of valid and reliable scoring. While this monitoring involves scoring staff, test development staff content coordinators, and psychometricians, those on the “front lines” are the trainer and supervisor who are on every team. Trainers are responsible for working with the scoring supervisor to ensure that backreading takes place at the specified rate and that backreading is conducted in a careful and thoughtful manner. Trainers are responsible for ensuring that the team does not drift from the appropriate scoring standards, using such methods as calibration sets and the judicious use of wall charts and other aids to keep scorers on track. Trainers are responsible for making sure that team interrater agreement remains high and for working with individual scorers when interrater agreement statistics and backreading indicate that they are scoring inaccurately. Trainers also have particular responsibility for the monitoring of cross-year scoring agreement.

Supervisors also play a crucial role in ensuring the quality of NAEP scoring. While trainers take the lead in teaching scorers how to score particular items and in making scoring decisions, experienced scoring supervisors provide critical content and technical support to trainers; they are expected to alert trainers or content leads to any issues encountered during backreading or other report-monitoring activities. Supervisors take the leading role in backreading, and therefore they must understand the items well and be able to analyze scoring errors. It is also vital that supervisors possess the ability to use monitoring features and functions of the scoring system.

When teams are having difficulties in scoring within the acceptable current-year or cross-year interrater agreement ranges, trainers consult with scoring staff or test development staff content coordinators. These content coordinators are staff members who have background in the subject being scored and substantial experience in NAEP scoring. They work with trainers to identify the source of the problems (e.g., whether training strategies can be improved, whether particular scorers need more guidance).

Printer-friendly Version