Comparing Private Schools and Public Schools Using Hierarchical Linear Modeling

July 2006

Authors: Henry Braun, Frank Jenkins, and Wendy Grigg

Download the complete report in a PDF file for viewing and printing. (885K PDF)

Executive Summary

Results from Grade 4
Results from Grade 8
Comparison of Results for Grade 4 and Grade 8
Cautions in Interpretation
Summary

A photograph of school-age children from the cover of the report.

The goal of the study was to examine differences in mean National Assessment of Educational Progress (NAEP) reading and mathematics scores between public and private schools when selected characteristics of students and/or schools were taken into account. Among the student characteristics considered were gender, race/ethnicity, disability status, and identification as an English language learner. Among the school characteristics considered were school size and location, and composition of the student body and of the teaching staff. In particular, if the student populations enrolled in the two types of schools differed systematically with respect to background characteristics related to achievement, then those differences would be confounded with straightforward comparisons between school types.

The present report examined results from the 2003 NAEP assessments in reading and mathematics for grades 4 and 8. NAEP draws nationally representative samples of schools and students. In 2003, over 6,900 public schools and over 530 private schools participated in the grade 4 assessments. Over 5,500 public schools and over 550 private schools participated in the grade 8 assessments.

Hierarchical linear models (HLMs) were employed to carry out the desired adjustments. HLMs were a natural choice because they accommodate the nested structure of the data (i.e., students clustered within schools) and facilitate the inclusion of variables derived from student and school characteristics. In this study, the focal parameter was the mean difference between mean NAEP scores for two populations of schools. (This difference was not identical to the difference in mean scores between the two student populations, though the discrepancy was typically small.) HLMs were used to compare all private schools to all public schools, as well as to compare, separately, certain categories of private schools (i.e., those for which sample sizes were sufficient to report reliable estimates) to all public schools. Statistical significance was determined at the .05 level using t tests on model results.

Results from Grade 4

Reading

In the first set of analyses, all private schools were compared to all public schools. The average private school mean reading score was 14.7 points higher than the average public school mean reading score, corresponding to an effect size of .41 (the ratio of the absolute value of the estimated difference to the standard deviation of the NAEP fourth-grade reading score distribution). After adjusting for selected student characteristics, the difference in means was near zero and not significant. In the second set of analyses, Catholic schools and Lutheran schools were each compared to all public schools. The results, both with and without adjustments, were similar to the corresponding results for all private schools.

Mathematics

In the first set of analyses, all private schools were again compared to all public schools. The average private school mean mathematics score was 7.8 points higher than the average public school mean mathematics score, corresponding to an effect size of .29. After adjusting for selected student characteristics, the difference in means was -4.5 and significantly different from zero. (Note that a negative difference implies that the average school mean was higher for public schools.) In the second set, Catholic schools and Lutheran schools were each compared to all public schools. The results, both with and without adjustments, were similar to the corresponding results for all private schools.

Results From Grade 8

Reading

In the first set of analyses, all private schools were compared to all public schools. The average private school mean reading score was 18.1 points higher than the average public school mean reading score, corresponding to an effect size of .58. After adjusting for selected student characteristics, the difference in means was 7.3 points and significantly different from zero. In the second set, Catholic, Lutheran, and Conservative Christian schools were each compared to all public schools. The results, both with and without adjustments, were generally similar to the corresponding results for all private schools. The only exception was that the average difference in adjusted school mean scores between Conservative Christian schools and all public schools was not significantly different from zero.

Mathematics

In the first set of analyses, all private schools were again compared to all public schools. The average private school mean mathematics score was 12.3 points higher than the average public school mean mathematics score, corresponding to an effect size of .38. After adjusting for selected student characteristics, the difference in means was nearly zero and not significant. In the second set, Catholic, Lutheran, and Conservative Christian schools were each compared to all public schools. While the results for Catholic schools, both with and without adjustments, were very similar to the corresponding results for all private schools, the results for the other two types differed.

The initial difference between Lutheran schools and all public schools was substantially larger (19.5 points) than was the case for all private schools. The average difference in adjusted mean mathematics scores between the two types of schools was 4.9 points and significantly different from zero. On the other hand, the initial difference between Conservative Christian schools and all public schools was substantially smaller (5.1 points) and not significant. The average difference in adjusted school means between Conservative Christian schools and all public schools was -7.6 points (i.e., a higher average school mean for public schools) and was significantly different from zero.

Comparison of Results for Grade 4 and Grade 8

Overall, there were many similarities in the results for the two grades. In both reading and mathematics, analyses employing unadjusted NAEP scores indicated that the average private school mean score was higher than the average public school mean score, and the difference was statistically significant. Including selected student characteristics in the model, however, resulted in a substantial reduction in the difference in all four analyses. The reduction varied from 11 to 15 score points. For grade 4 reading and grade 8 mathematics, the average difference in adjusted school mean scores was no longer significant. For grade 4 mathematics, the difference was significant, and the adjusted school mean was higher for public schools. Only for grade 8 reading was the difference still significant with a higher school mean for private schools. For all four analyses, with student characteristics such as gender and race/ethnicity incorporated in the model, the inclusion of school characteristics (e.g., teacher experience, type of school location, school size) had little impact on the estimate of the average difference between the two types of schools.

Variance decompositions yielded similar results for the four grade-subject combinations. Most of the total variance was due to heterogeneity among students within schools rather than heterogeneity among school mean scores. The combination of selected student and school characteristics accounted for about one-third of the total variance for grade 4 and about two-fifths of the total variance for grade 8.

Cautions in Interpretations

When interpreting the results from any of these analyses, it should be borne in mind that private schools constitute a heterogeneous category and may differ from one another as much as they differ from public schools. Public schools also constitute a heterogeneous category. Consequently, an overall comparison of the two types of schools is of modest utility. The more focused comparisons conducted as part of this study may be of greater value. However, interpretations of the results should take into account the variability due to the relatively small sizes of the samples drawn from each category of private school, as well as the possible bias introduced by the differential participation rates across private school categories.

There are a number of other caveats. First, the conclusions pertain to national estimates. Results based on a survey of schools in a particular jurisdiction may differ. Second, the data are obtained from an observational study rather than a randomized experiment, so the estimated effects should not be interpreted in terms of causal relationships. In particular, private schools are “schools of choice.” Without further information, such as measures of prior achievement, there is no way to determine how patterns of self-selection may have affected the estimates presented. That is, the estimates of the average difference in school mean scores are confounded with average differences in the student populations, which are not fully captured by the selected student characteristics employed in this analysis.

Summary

In grades 4 and 8 for both reading and mathematics, students in private schools achieved at higher levels than students in public schools. The average difference in school means ranged from almost 8 points for grade 4 mathematics, to about 18 points for grade 8 reading. The average differences were all statistically significant. Adjusting the comparisons for student characteristics resulted in reductions in all four average differences of approximately 11 to 14 points. Based on adjusted school means, the average for public schools was significantly higher than the average for private schools for grade 4 mathematics, while the average for private schools was significantly higher than the average for public schools for grade 8 reading. The average differences in adjusted school means for both grade 4 reading and grade 8 mathematics were not significantly different from zero.

Comparisons were also carried out with subsets of private schools categorized by sectarian affiliation. After adjusting for student characteristics, raw score average differences were reduced by about 11 to 15 points. In grade 4, Catholic and Lutheran schools were each compared to public schools. For both reading and mathematics, the results were generally similar to those based on all private schools. In grade 8, Catholic, Lutheran, and Conservative Christian schools were each compared to public schools. For Catholic and Lutheran schools for both reading and mathematics, the results were again similar to those based on all private schools. For Conservative Christian schools, the average adjusted school mean in reading was not significantly different from that of public schools. In mathematics, the average adjusted school mean for Conservative Christian schools was significantly lower than that of public schools.

Download the complete report in a PDF file for viewing and printing. (885K PDF)

NCES 2006-461 Ordering information

Suggested Citation
Braun, H., Jenkins, F., and Grigg, W. (2006). Comparing Private Schools and Public Schools Using Hierarchical Linear Modeling (NCES 2006–461). U.S. Department of Education, National Center for Education Statistics, Institute of Educational Sciences. Washington, DC: U.S. Government Printing Office.

For more information, see Private Schools in America in the Special Studies section of this website.

Last updated 04 August 2006 (RF)