Skip Navigation

Student Participation in Community Service Activity / Chapter 5



Survey Methodology and Data Reliability

The 1996 National Household Education Survey (NHES:96) is a telephone survey conducted by Westat for the U.S. Department of Education, (NCES). Data collection took place from January through April of 1996. When appropriately weighted, the sample is nationally representative of all civilian, non institutionalized persons in the 50 states and the District of Columbia. The sample was selected using random-digit-dialing (RDD) methods, and the data were collected using computer-assisted telephone interviewing (CATI) technology. See Vaden-Kiernan et al. (forthcoming) for more information.

The Youth Civic Involvement (CI) component of the NHES:96, which is the primary basis of this report, employed a sample of students in grades 6 through 12. Up to three instruments were used to collect information on the school and family experiences of these students. A set of screening items (Screener), administered to a member of the household age 18 or older, was used to determine whether any children of the appropriate ages or grades lived in the household, to collect information on each household member, and to identify the appropriate parent/guardian respondent for the sampled child. For sampling purposes, children residing in the household were grouped into younger children (age 3 through grade 5) and older children (grades 6 through 12). If one child age 3 to 5th grade resided in the household, an interview was conducted about that child; if there were multiple children in this range, one child was sampled with equal probability. Similarly, if one child in 6th through 12th grade resided in the household, an interview was conducted about that child; if there were multiple children in this range, one child was sampled with equal probability. Up to two children could have been selected from a household, one in the lower age/grade range and one in the higher grade range.

For households with youth in 6th through 12th grade who were sampled for the survey, a Parent and Family Involvement in Education/Civic Involvement (PFI/CI) interview was conducted with the parent/guardian most knowledgeable about the care and education of the youth, usually the child's mother. Following completion of that interview and receipt of parental permission, a Youth CI interview was conducted with the student. This report was based on the responses of these students, and with few exceptions all variables used in the report are on the Youth CI public data file. The analysis includes one item taken from the Parent PFI/CI data file: the student's academic performance. Two variables were created from the 1990 Decennial Census (STF 3B data file) by matching the respondent's ZIP code to the census data. These items are the percentage of households below poverty and the percentage of households that are owner occupied. These variables are contained in the NHES:96 Household & Library restricted data file, which can be obtained from NCES under a special licensing agreement. More information about the adult, parent, and youth data can be found in the National Household Education Survey of 1996, Data File User's Manual, Volumes I-V (Collins et al. forthcoming).

Response Rates

For the NHES:96 survey, Screeners were completed with 55,708 households. A sample of 23,835 children age 3 through 12th grade was selected for a Parent PFI/CI interview. This sample included 10,949 youth in grades 6 through 12. The response rate for the Screener was 70 percent. The completion rate for the Parent PFI/CI interview, or the percentage of eligible sampled children for whom interviews were completed, was 89 percent, or 20,792 interviews. Thus, the overall response rate for the Parent PFI/CI interview was 63 percent (the product of the Screener response rate and the Parent PFI/CI completion rate). An interview with a sampled youth was attempted only after the interview with his or her parent had been completed. The completion rate for youth in grades 6 through 12 was 76 percent. Thus, the overall response rate for the Youth CI interview was 53 percent (the product of the Screener completion rate and the Youth CI interview completion rate). For more information about response rates, see Montaquila and Brick (forthcoming). This report is based on a subset of the total population of youth, students enrolled in schools in the 6th through 12th grade. The unweighted number of cases included in this analysis is 7,940.

For the NHES:96, item non response (the failure to complete some items in an otherwise completed interview) was very low. Most items in used in this analysis have response rates of 98 percent or more as shown in exhibit 1. Items in this report that had a response rate of less than 98 percent are number of weeks and number of hours per week the student performs community service, whether the school arranges and/or requires student community service, whether the student will participate in community service later this year or next year, whether the student participated in student government, and household income. Through a procedure known as "hot-deck" imputation (Kalton and Kasprzyk 1986), responses were imputed for missing values (i.e., "don't know" or "refused," for items not specifically designated to have those as legitimate response categories, or "not ascertained"). As a result, no missing values remain.

Exhibit 1.-Item response rates for variables used in the analysis

-----------------------------------------------------------------------------
                                                                      Item
                                                          Number    response
Variable      Label                                      eligible     rate
-----------------------------------------------------------------------------
SEX           S6-GENDER AT SCREENER                       8,043      99.98%
RACE          SX21-RACE                                   8,043      99.52%
HISPAN        SX22-HISPANIC                               8,043      99.45%
HOWNHOME      SX27-OWN, RENT HOME/OTHR ARRNGMNT           8,043      99.88%
HINCMRNG      SX33- TOTAL HH INCOME RANGE                 8,043      94.89%
HINCOME       SX33- TOTAL HH INCOME RANGE 2               8,043      91.35%
CSPEAK        PA3-LANG CHLD SPEAKS MOST AT HOME          20,792     100.00%
GRADE         PB4-GRADE/YR CHLD IS ATTENDING             19,135     100.00%
SPUBLIC       PD1-CHLD ATTNDS PUBL/PRIV SCH              19,343      99.86%
SCHOICE       PD3-SCH ASSIGNED OR CHOSEN                  7,130      99.99%
SEGRADES      PE3-CHLD'S GRADES ACROSS ALL SUBJECTS      16,151      99.00%
MOMGRADE      PL3-HIGHEST GRADE MOM COMPLETED            20,026      99.29%
DADGRADE      PM3-HIGHEST GRADE DAD COMPLETED            15,825      98.75%
PRREPGOV      YB2-SERVED/WORKED IN STUDENT GOVT           6,494      91.98%
PRSCHACT      YB3-PARTICIPATED IN SCH ACTIVITIES          7,940      99.87%
PRGRPACT      YB4-PARTICIPATED OUT-OF-SCH ACTIVITIES      8,043      99.93%
PRWORK        YB5-WORKS FOR PAY                           8,043      99.91%
SACTY         YC1-DOES COMMTY SERVICE ACTY                8,043      99.69%
SAREG1        YC4-SERVICE ACTIVITY #1 SCHEDULE            3,996      99.60%
SAWKS1        YC5-FREQ OF SERVICE ACTIVITY #1             1,717      96.85%
SAWKSNU1      YC5OV-NUM WKS FOR SERV ACTY #1              1,162      96.21%
SAHRS1        YC6-HRS/WK FOR SERV ACTY #1                 1,717      97.32%
SAHRSNU1      YC6OV-NUM HRS/WK FOR SERV ACTY #1           1,691      96.69%
SAREG2        YC4-SERVICE ACTIVITY #2 SCHEDULE            1,557      99.17%
SAWKS2        YC5-FREQ OF SERVICE ACTIVITY #2               623      97.75%
SAWKSNU2      YC5OV-NUM WKS FOR SERV ACTY #2                434      97.70%
SAHRS2        YC6-HRS/WK FOR SERV ACTY #2                   623      98.56%
SAHRSNU2      YC6OV-NUM HRS/WK FOR SERV ACTY #2             606      98.02%
SAREG3        YC4-SERVICE ACTIVITY #3 SCHEDULE              458      99.13%
SAWKS3        YC5-FREQ OF SERVICE ACTIVITY #3               217      95.85%
SAWKSNU3      YC5OV-NUM WKS FOR SERV ACTY #3                140      97.14%
SAHRS3        YC6-HRS/WK FOR SERV ACTY #3                   217      96.77%
SAHRSNU3      YC6OV-NUM HRS/WK FOR SERV ACTY #3             211      95.73%
SAARRSER      YC8-SCH ARRANGES SERV ACTIVITIES            7,940      93.61%
SAREQSER      YC9-SCH REQUIRES SERV ACTY                  7,940      92.49%
SATALK        YC11-TALK IN CLASS/GRP ABT SERV ACTY        3,956      99.32%
SAJOURNAL     YC12-REQUIRED TO WRITE ABT SERV ACTY        3,956      99.54%
SAGRADE       YC13-ACTIVITY FOR A GRADE IN CLASS          3,956      99.04%
SASCHLYR      YC14-WILL DO SERV ACTY LATER THIS SCH YR    4,047      85.37%
SANEXTYR      YC15-WILL DO SERV ACTY NEXT YR              8,043      87.87%
SASERVC       YC17-FAM PARTICIPATES COMMTY SERV           8,043      98.45%
-----------------------------------------------------------------------------        
NOTE: The following variables were also used in the report analyses: CENREG, ZIP18POV, and ZIPOWNED. These were not questionnaire variables but rather variables derived from respondents' telephone area codes or ZIP codes.

Data Reliability

Estimates produced using data from the NHES:96 are subject to two types of error, non sampling and sampling errors. Sampling errors occur because the data are collected from a sample rather than a census of the population. non sampling errors are errors made in the collection and processing of data.

Weighting and sampling errors

All of the estimates in the report are based on weighting the observations using the probabilities of selection of the respondents and other adjustments to partially account for non response and coverage bias. These weights were developed to make the estimates unbiased and consistent with estimates of the national totals. There is a potential for bias in the estimates due to the high non response in this survey. Analyses of response rates for different classifications of the sampled youth also demonstrated differential response rates according to the age and grade of child. To reduce potential non response bias, grade was used in the construction of weighting classes for non response adjustment. For more information about adjustment for non-response, see Montaquila and Brick (forthcoming).

The sample of telephone households selected for the NHES:96 is just one of many possible samples that could have been selected. Therefore, estimates produced from the NHES:96 sample may differ from estimates that would have been produced from other samples. This type of variability is called sampling error because it arises from using a sample of households with telephones, rather than all households with telephones.

The standard error is a measure of the variability due to sampling when estimating a statistic. Standard errors can be used as a measure of the precision expected from a particular sample. The probability that a complete census count would differ from the sample estimate by less than 1 standard error is about 68 percent. The chance that the difference would be less than 1.65 standard errors is about 90 percent, and that the difference would be less than 1.96 standard errors, about 95 percent.

In addition to properly weighting the responses, special procedures for estimating the statistical significance of the estimates were employed because the data were collected using a complex sample design. Complex sample designs, like that used in the NHES, result in data that violate some of the assumptions that are normally required to assess the statistical significance of the results. Frequently, the sampling errors of the estimates from the survey are larger than would be expected if the sample was a simple random sample and the observations were independent and identically distributed random variables.

Replication methods of variance estimation were used to reflect the actual sample design used in the NHES:96. A form of the "jackknife" replication method (Wolter 1985) was used to compute the standard errors for estimates presented in this report. The jackknife methods were used to estimate the precision of the estimates of the reported national totals, percentages, and regression parameters. The idea behind replication methods is to form subsamples, or replicates, and then calculate the estimate of interest from the full sample as well as each replicate. The variation among the replicate estimates is used to estimate the variance for the full sample. To form the replicates for the jackknife method used in this analysis, 80 subsets of telephone numbers were identified. These subsets were constructed to be approximately equal in size, with each subset resembling the full sample. Replicates were formed by deleting one subset at a time and adjusting the weights of units (households or persons) in the other subsets accordingly.

Standard errors for all of the estimates are presented. These standard errors can be used to produce confidence intervals. For example, an estimated 26 percent of students reported regular participation in a service activity. This figure has an estimated standard error of 0.6. Therefore, a 95 percent confidence interval for the percentage of students reporting regular participation in a service activity is approximately 25 to 27 percent.

To test the differences between two categories (e.g., 6th through 8th graders versus 9th and 10th graders), Student's t statistic was employed, using unbiased estimates of sampling errors derived by the replication methods mentioned above. As the number of comparisons at the same significance level increases, it becomes more likely that at least one of the estimated differences will be significant merely by chance, that is, it will be erroneously identified as different from zero. Even when there is no statistical difference between the means or percentages being compared, there is a 5 percent chance of getting a significant F or t value due to sampling error alone. As the number of comparisons increases, the chance of making this type of error also increases.

A Bonferroni adjustment was used to correct significance tests for multiple comparisons. This method adjusts the significance level for the total number of comparisons made with a particular classification variable. All the differences cited in this report are significant at the 0.05 level of significance after a Bonferroni adjustment.

non sampling errors

non sampling error is the term used to describe variations in the estimates that may be caused by population coverage limitations and data collection, processing, and reporting procedures. The sources of non sampling errors are typically problems like unit and item non response, the differences in respondents' interpretations of the meaning of the questions, response differences related to the particular time the survey was conducted, and mistakes in data preparation. As explained above, weighting procedures help to reduce potential bias due to non response.

In general, it is difficult to identify and estimate either the amount of non sampling error or the bias caused by this error. In the NHES:96, efforts were made to prevent such errors from occurring and to compensate for them where possible. For instance, during the survey design phase, focus groups and cognitive laboratory interviews were conducted for the purpose of assessing respondent knowledge of the topics, comprehension of questions and terms, and the sensitivity of items. The design phase also entailed CATI instrument testing and an extensive, multi-cycle field test in which about 3,200 Screeners, over 950 parent interviews, about 300 youth interviews, and about 40 adult interviews were conducted.

An important non sampling error for a telephone survey is the failure to include persons who do not live in households with telephones. About 93.3 percent of all students in grades 1 through 12 live in households with telephones. Estimation procedures were used to help reduce the bias in the estimates associated with youth who do not live in telephone households. Cross-classifications of race/ethnicity by household income, census region by urbanicity, and home tenure by child's grade were used for forming cells for raking. For more information about coverage issues and estimation procedures, see Brick and Burke (1992) and Montaquila and Brick (forthcoming).

Hours of Service Measure

The measure presented in this report for the number of hours of community service in which a student had participated was developed by combining information about the number of weeks and the number of hours per week that students reported spending in each of up to three service activities. First, the number of weeks that the student had participated in each activity was calculated. The exact number of weeks was used in the calculation if it was reported. For students who reported participating since the beginning of the school year, the number of days from September 1, 1995, to the date of the interview was calculated and divided by 7 to obtain the number of weeks. Some students (fewer than 3 percent for any service activity) responded in some other way (for example, "three times a month"). For service activity one, these cases were assigned the modal value for service activity one; that is, they were given the most frequently reported number of weeks for the first service activity named by students. The same procedure was used to assign the number of weeks for service activity two and service activity three.

Second, the number of hours for each service activity was calculated. If a specific number of hours had been reported, that number was used. For the few students who gave another response (e.g., "the hours change from week to week"), the modal value for number of hours for the appropriate service activity (first, second, or third) was assigned. Modal values were assigned to less than 3 percent of students in any given activity.

Third, the total number of hours of service in the current school year was calculated. The number of hours per week was multiplied by the number of weeks for each service activity. The total number of hours of community service for each student was calculated by summing the hours for each of the three possible service activities.

Students' Expectations of Serving Later in the School Year

To examine whether or not students actually met their expectations to do community service later in the school year, students' responses were grouped according to which month they responded to the survey. If students were fulfilling their plans of getting involved in community service, one would expect that the percentage saying they planned to participate would decrease as time progressed, both because students at later times would have had more opportunities to participate, and because there would be less chance to still participate later in the school year. The actual change was fairly small. Among students who responded over the period January 2 through February 2, 33 percent said they had not yet participated but would participate later in the school year, while among students in the last 6 weeks of data collection (March 3 through April 13), 30 percent said they would participate later. Further, the insignificant decline in the percentage who said they would participate later was not matched by an increase in the number who had participated, but rather in the percentage who said they had no plans to participate in community service during the school year. There is thus some reason for thinking that many of the students who said they planned to participate still would not have participated in community service by the time the school year was completed.[11]

Multivariate Analysis

To design the multivariate analysis presented in table 11, all variables that were statistically significant in the bivariate analyses were included in a logistic regression. Several variables, however, were no longer significant when a multivariate model was applied (e.g., number of parents in the household). Such variables could still be important (e.g., significance depends in part on the number of cases available for analysis). To provide a clearer picture of the significant variables, most of the variables that were not significant in the full model were dropped from the analysis. The important exception was the school service requirement variable. This variable was kept in the model because one of the main purposes of the analysis was to study whether or not arranging service and requiring service each had significant effects. For completeness, table 13 presents the full model before variables were excluded. It demonstrates that the inclusion or exclusion of these extra variables did not have an important effect on the estimates.

Table 13.-Logistic regression analysis to predict student participation in community service (full model, including variables that were not statistically significant)

----------------------------------------------------------------------------------------------------------
                                               Coefficient/1/   Standard error    P-value   Odds ratio/2/
----------------------------------------------------------------------------------------------------------
Intercept                                          -3.12**           0.180         0.000       0.04**
Student characteristics
Female                                               0.14*           0.052         0.010        1.15*
English spoken most at home by student              0.39**           0.127         0.003       1.48**
Number of types of student activities               0.60**           0.037         0.000       1.82**
Student in grades 11 or 12                          0.21**           0.066         0.002       1.23**
Student received mostly As                          0.50**           0.079         0.000       1.65**
Student received mostly Bs                           0.20*           0.079         0.013        1.22*
White                                                 0.08           0.082         0.356         1.08
Family characteristics
Adult performed community service                   0.32**           0.066         0.000       1.38**
Parent/guardian had college degree or higher        0.25**           0.073         0.001       1.28**
High-income household                                 0.01           0.075         0.882         1.01
Number of parents in household                       -0.02           0.068         0.730         0.98
School characteristics
Church-related school                               0.58**           0.136         0.000       1.79**
Other private school                                  0.13           0.230         0.586         1.14
School policies
School required service                              -0.39           0.302         0.206         0.68
School arranged service                             0.95**           0.093         0.000       2.59**
School both required and arranged service             0.55           0.318         0.090         1.73
Community characteristics
Percent of households in ZIP code owner occupied
   60 percent or less                                -0.04           0.076         0.615         0.96
   61 to 70 percent                                   0.05           0.094         0.570         1.05
Percent of families below poverty in ZIP code
   Less than 5 percent                               -0.09           0.117         0.427         0.91
   5 to 9 percent                                     0.03           0.128         0.798         1.03
   10 to 19 percent                                   0.01           0.109         0.930         1.01
----------------------------------------------------------------------------------------------------------    
*p<.05. **p<.01.

1\ These coefficients differ slightly from those in table 11 because a different model is being used. Each category is expressed in relation to an omitted category for the variable, controlling for all other variables in the model. The applicable coefficients can be summed to estimate the probability that students with certain characteristics will participate in community service. For example, for a male student who spoke English the most at home, who participated in four other types of activities, who received mostly Bs, who had an adult in the household who performed community service, who attended a church-related private school, and who fit none of the other criteria, the sum is -3.12 + .39 +4(.60) + .20 + .32 + .58 = .77. Based on these characteristics, the probability of participation is 1/(1 + e-.77) = .68, or 68 percent.
2\ The odds ratio can be used to estimate the change in probability of a student participating in community service. An odds ratio greater than one indicates that students in the indicated group were more likely to perform community service than the omitted group. More specifically, suppose a student would ordinarily have a probability of participating of 68 percent, but that student is in a school that arranges service. The student's original probability can be expressed as an odds [68/(100-68) = 68/32 = 2.125]. The odds ratio of 2.59 for schools that arrange service can be multiplied by the original odds (2.59 times 2.125 = 5.50) to estimate the revised odds that the student would participate, based on the school's policy. To express the revised odds as a probability, one can apply the formula: probability = (odds)/ (1 + odds) = 5.50/6.50 = .85. Thus, by being in a school that arranges services, the student's probability of participating would increase from 68 percent to 85 percent. The amount of the increase that is associated with the school policy varies from one student to another depending on the student's original probability of participating.

SOURCE: U.S. Department of Education, , National Household Education Survey, spring 1996, Youth Civic Involvement component.


Footnotes:

[11] Students' expectations of serving later in the school year might have greater accuracy in at least one situation. The survey focuses on 6th through 12th graders, but in the case of those 12th graders who were required to participate in community service in order to graduate, the graduation requirements may help to ensure that the expectations were met (though the students could have satisfied the requirement in an earlier year). To examine this possibility, the responses of 12th graders were looked at separately. The data indicate that 12th grade students who attended schools in which students were required to participate in community service tended to have already participated in community service earlier in the school year; only a small group of all of the students who said they expected to participate later actually met the conditions of also being 12th graders who were required to participate. Further, the same pattern appeared for these students as for students overall, with students who responded later in the school year not showing increased rates of participation, but rather showing small (but not statistically significant) increases in the rate of non participation


Chapter 4. Summary and ConclusionsPrev Contents NextReferences