Technical Report:

NELS:88 Survey Item Evaluation Report

March 1997

Highlights

This report is designed for researchers who may use the NELS:88 data to study the longitudinal progress of eighth graders through their next four years. Its purpose is to convey information about the measurement characteristics of the survey items in the base year and first and second follow-ups—in particular, about the nature of nonresponse and about convergence of information from alternative sources. The analysis of measurement characteristics is necessary because substantive research depends on the ability to generalize from responses to items on a paper-and- pencil form to tests of models and theories about how schools work and students learn. Percentages of responses depend on both the state of the individual (the so-called “true value”) and the processes by which the individual interprets the item wording and decides which of several responses to make. All of the items in NELS:88 are multiple choice (bubble-coded) items, and the wording of both the question and the response alternatives affects the response generation process.

Nine separate sources of NELS:88 information are covered in this report: student responses in the base year, first follow-up, and second follow-up; dropout responses in the first and second follow-ups; parent responses in the base year and second follow-up; teacher responses at the second follow-up; and transcript records. Even if questions are worded identically and understood identically by different respondents, discrepancies emerge because each individual has unique knowledge and a unique perspective on the situation specified by the item (e.g., “Is the school safe?”). These discrepancies can be interpreted in several different ways, and in most cases, data are not sufficient to determine which interpretation is more accurate.

One interpretation of discrepancies, or lack of convergence of responses to the same item (about the same student) from two sources, is in terms of reliability. If we assume that responses from two “judges” are “measuring” the same “construct,” then the correlation of responses between judges indicates the extent to which the construct is being reliably measured. It is often called the inter-judge reliability. If the two responses are from the same judge at two different points in time, the correlation indicates the test-retest reliability. Underlying this interpretation is the assumption that the two responses are each measuring the same construct, each with some “measurement error” that leads to discrepancies.

A second interpretation of convergence is in terms of validity. If we assume that one of the measures is an “accurate” (or criterion) measure of a construct and the other is an indicator or a predictor of that construct, then the correlation of responses indicates the validity of the predictor. The concept of validity is employed when carrying out research in which one measure (the predictor) is available but the other (the construct) is not. For example, eighth grade achievement scores might be considered predictors of twelfth grade achievement scores, and high school grades and test scores might be considered predictors of success in college. Underlying this interpretation is the assumption that variation in predictor values is a sum of (a) variation on the underlying construct and (b) measurement error. The square of the correlation coefficient indicates the proportion of the variance in the predictor that reflects variation in the underlying construct; however, the assumption that an observed “criterion” is identical to (or a perfect measure of) the construct of interest is often relaxed in practice, and validity coefficients are adjusted (upward) for criterion unreliability, or attenuation.

The third interpretation of discrepancies is in terms of communality between separate constructs, each measured by a questionnaire response. Two discrepant responses to the same item about the same subject may both be reliable and valid—but they are measuring different things, and the phenomenon of their discrepancy can be a topic for substantive psychological or sociological research. In fact, lack of communality between apparently similar measures can provide new insights into processes under study.

For example, in NELS:88, indicators of student disability were obtained from students, parents, teachers, and school officials. Each indicator was worded differently, but all were ostensibly related to an underlying construct of student disability. Results of comparisons of these measures showed that there was very little overlap (far less than 50 percent) in the population of students identified as disabled by these separate sources (Rossi and Wolman 1996). Rather than interpret this phenomenon as an indicator of unreliability or lack of validity, Rossi et al. interpreted the results in terms of different item wordings and the different perspectives that students, teachers, and parents have on a student's disability, developing a multidimensional picture of disability of high school students.

Other examples abound. To the extent that teenagers and their parents provide different reports on the frequency of non-English language use in the home, both may be accurately viewing the same language use, but from different contexts. To the extent that teenagers and teachers differ on the extent to which they report that students do experiments in class, both may be accurately viewing the same classroom experiences, but from different contexts. To the extent that a student reports different expectations for college graduation between base year and second follow-up surveys, the base year measure may capture more variation related to choices the student makes in ninth and tenth grades (e.g., course selection), while the second follow-up measure captures more variation related to student achievement during high school (e.g., GPA, test scores).

To decide among these interpretations requires an independent source of information—either a separate, accurate measure of the reliability of the items or a logical argument that one of the measures can be treated as identical to the construct (i.e., as a criterion) for a researcher's purposes. This information is generally not available for measures in NELS:88 or any other survey. However, that is not critical for the value of information about convergence: researchers who might use NELS:88 for substantive research on educational policy and practice can take lack of convergence into account in their interpretations of results, no matter which of the explanations of discrepancies is accurate. The approach taken in this report is to describe the extent of discrepancies (or lack of convergence), and in some cases to suggest possible reasons for discrepancy patterns.

Download/view the full report in a PDF file.(1,390K)

Help with PDF files

For more information about the content of this report, contact Ralph Lee at Ralph.Lee@ed.gov.