Skip Navigation
small NCES header image
NAEP Instruments → Cognitive Items and Instruments

Cognitive Items and Instruments

       

Student Booklet Block Design
Student Booklet Spiral Design
Bundling of the Student Booklets

horizontal line

NAEP assessments include cognitive items, test questions that are designed to assess what students know and can do, and background questions. Cognitive items are based on the framework and specifications documents for each assessment subject. These types of items include multiple-choice items, constructed-response items scored dichotomously, and constructed-response items scored polyomously. Background questions gather information such as time spent studying. NAEP subject areas include the arts, civics, economics, foreign language, geography, mathematics, reading, writing, science, U.S. history or world history.

The item-development steps for each subject area are as follows:

  1. The National Assessment Governing Board provides content frameworks and item specifications in each subject area.

  2. Instrument development committees in each subject area provide guidance to NAEP staff about how the objectives can be measured given the constraints of resources and the feasibility of measurement technology. The committees make recommendations about priorities for the assessment (within the context of the assessment framework) and the types of items to be developed.

  3. Items are chosen for the assessment through an extensive selection process that involves the input of practitioners from across the country and members of the instrument development committees.

  4. Specialists with subject-matter expertise and experience in creating items according to specifications develop and review the assessment questions.

  5. The items and accompanying scoring guides are reviewed and revised by NAEP test development staff and external test specialists.

  6. Representatives from the state education agencies meet and review all items and background questionnaires that are scheduled to be part of the state assessment.

  7. Editorial and fairness reviews are conducted as required by NCES.

  8. Field test materials are prepared, including those necessary to secure clearance by the federal Office of Management and Budget.

  9. A field test is conducted in many of the states and jurisdictions slated to participate in the following year's operational assessment.

  10. Representatives from state education agencies meet and review the field test results for all items selected for the state assessment.

  11. Based on the field test analyses, new items for each subject area are revised or modified where necessary. The items once again undergo the full range of reviews.

  12. Each subject-area instrument development committee approves the selection of items to include in the following year's operational assessment.

  13. After a final review and a check to ensure that each assessment booklet and each block meet the overall guidelines for the assessment, the booklets are typeset and printed.

1 Item development steps for the background questionnaires are different and are explained in the background questionnaire section.

Each administration of the NAEP assessment requires a new design of the instruments given to students—the student booklets. To allow for wide content coverage with limited testing time for each student, the instrument scheme requires a three-step design process for the subject areas to be assessed:

  • student booklet block design,
  • student booklet spiral design, and
  • bundling of the student booklets.

In the first step, the booklet block designs are created so that no student receives too many items and all receive interlocking blocks of items in a focused balanced, or partially balanced incomplete block (BIB or pBIB) design. The focused BIB or pBIB design allows for improved estimation within a particular subject area, and estimation continues to be optimized for groups rather than individuals.

Second, the spiraling scheme is designed. Spiraling refers to interleaving booklets systematically so that when they are handed out in the specified order, any group of students will receive approximately equal proportions of different types of booklets.

The third aspect is the bundling design. NAEP test developers created and implemented an enhanced bundling design in 2003, referred to as vertical bundling. Vertical bundling has flexibility with respect to bundle length and reduces the required number of different bundles, decreasing booklet wastage, and improving balance of within-session booklet pairings.

Note: Until the 1984 assessment, NAEP was administered using matrix sampling and tape recorders; that is, by administering booklets of exercises using paced audio tapes that walked groups of students through the individual assessment exercises in a common booklet. In the 1984 assessment, a balanced incomplete block booklet design, which does not include audio tape pacing, was introduced in place of taped matrix sampling.

Last updated 14 March 2011 (GF)
Would you like to help us improve our products and website by taking a short survey?

YES, I would like to take the survey

or

No Thanks

The survey consists of a few short questions and takes less than one minute to complete.
National Center for Education Statistics - http://nces.ed.gov
U.S. Department of Education