SUBJECT: PRETESTING SURVEY SYSTEMS

NCES STANDARD: 2-4

PURPOSE: To ensure that all components of a survey system will function as intended when implemented in the full-scale survey.

KEY TERMS: edit, estimation, field test, frame, imputation, instrument, pretest, response rates, stage of data collection, survey system, and variance.

STANDARD 2-4-1: One type of a pretest is a pilot test in which some components of a survey system can be pretested prior to a field test of the survey system (for example, focus groups, cognitive laboratory work, pilot tests, and or calibration studies).

STANDARD 2-4-2: A second type of pretest is a field test. Components of a survey system that cannot be successfully demonstrated through previous work must be field tested prior to implementation of the full-scale survey. The design of a field test must reflect realistic conditions, including those likely to pose difficulties for the survey. Documentation of the field test (e.g., materials for technical review panels, working papers, technical reports) must include the design of the field test; a description of the procedures followed; analysis of the extent to which the survey components met the pre-established criteria; discussion of other potential problems uncovered during the field test; and recommendations for changes in the design to solve the problems.

GUIDELINE 2-4-2A: Elements to be tested and measured may include alternative approaches to accomplishing a particular task. Elements to be tested may include: frame development; sample selection; questionnaire design; data collection; response rates; data processing (e.g., entry, editing, imputation); estimation (e.g., weighting, variance computation); file creation; and tabulations.

GUIDELINE 2-4-2B: For an ongoing survey, new elements or content should be field tested, along with elements being changed as a result of the evaluation of the survey (see Standard 4-3).

GUIDELINE 2-4-2C: The evaluation criteria for a successful field test should be developed before the field test begins. Key evaluation criteria are established during the design stage. If the criteria are not met, that survey component should not be implemented without field testing a redesigned component.

GUIDELINE 2-4-2D: The results of a field test should be available and analyzed for internal use prior to making a decision to implement the full-scale survey.

GUIDELINE 2-4-2E: Survey design and instrumentation should be revised to reflect modifications suggested by the results of the field test. A revised budget should be developed, if necessary, to reflect both changes in design and knowledge gained during the field test about resource requirements.