​​​​​​NAEP Technical DocumentationReplicate Variance Estimation

Variances for NAEP assessment estimates are computed using the paired jackknife replicate variance procedure. This technique is applicable for common statistics, such as means and ratios, and differences between these for different subgroups, as well as for more complex statistics such as linear or logistic regression coefficients.

In general, the paired jackknife replicate variance procedure involves initially pairing clusters of first-stage sampling units to form \(H\) variance strata \(h = 1, 2, 3, ..., H\) with two units per stratum. The first replicate is formed by assigning, to one unit at random from the first variance stratum, a replicate weighting factor of less than 1.0, while assigning the remaining unit a complementary replicate factor greater than 1.0, and assigning all other units from the other \(H - 1\) strata a replicate factor of 1.0. This procedure is carried out for each variance stratum resulting in \(H\) replicates, each of which provides an estimate of the population total.

In general, this process is repeated for subsequent levels of sampling. In practice, this is not practicable for a design with three or more stages of sampling, and the marginal improvement in precision of the variance estimates would be negligible in all such cases in the NAEP setting. Thus in NAEP, when a two-stage design is used–sampling schools and then students–beginning in 2011 replication is carried out at both stages for the purpose of computing replicate student weights. The change implemented in 2011 permitted the introduction of a finite population correction factor at the school sampling stage. Prior to 2011, replication was only carried out at the first stage of selection. See Rizzo and Rust (2011) for a description of the methodology. 

When a three-stage design is used, involving the selection of geographic Primary Sampling Units (PSUs), then schools, and then students, the replication procedure is only carried out at the first stage of sampling (the PSU stage for noncertainty PSUs, and the school stage within certainty PSUs). In this situation, the school and student variance components are correctly estimated, and the overstatement of the between-PSU variance component is relatively very small.

The jackknife estimate of the variance for any given statistic is given by the following formula:

\begin{equation} \nu(\hat{t}) =\sum_{h=1}^{H} {(\hat{t}_{h}-\hat{t})^2}, \end{equation}

where

• \(\hat{t}\) represents the full sample estimate of the given statistic; and

• \(\hat{t}_{h}\) represents the corresponding estimate for replicate \(h\).

Each replicate undergoes the same weighting procedure as the full sample so that the jackknife variance estimator reflects the contributions to or reductions in variance resulting from the various weighting adjustments.

The NAEP jackknife variance estimator is based on 62 variance strata resulting in a set of 62 replicate weights assigned to each school and student.

The basic idea of the paired jackknife variance estimator is to create the replicate weights so that use of the jackknife procedure results in an unbiased variance estimator for totals and means, which is also reasonably efficient (i.e., has a low variance as a variance estimator). The jackknife variance estimator will then produce a consistent (but not fully unbiased) estimate of variance for (sufficiently smooth) nonlinear functions of total and mean estimates such as ratios, regression coefficients, and so forth (Shao and Tu 1995).

The development below shows why the NAEP jackknife variance estimator returns an unbiased variance estimator for totals and means, which is the cornerstone to the asymptotic results for nonlinear estimators. See for example Rust (1985). This paper also discusses why this variance estimator is generally efficient (i.e., more reliable than alternative approaches requiring similar computational resources).

The development is done for an estimate of a mean based on a simplified sample design that closely approximates the sample design for first-stage units used in the NAEP studies. The sample design is a stratified random sample with \(H\) strata with population weights \(W_{h}\), stratum sample sizes \(n_{h}\), and stratum sample means \(\overline{y}_{h}\). The population estimator \(\hat{\overline{Y}}\) and the standard unbiased variance estimator \(\nu(\hat{\overline{Y}})\) are

\begin{equation} \hat{\overline{Y}} =\sum_{h=1}^{H} W_{h}\overline{y}_{h}, \end{equation}

\begin{equation} \nu \left(\hat{\overline{Y}} \right) = \sum_{h=1}^{H} W_{h}^2 \frac{s_h^2}{n_{h}}, \end{equation}

\begin{equation} s^2_h=\frac{1}{n_{h}-1} \sum_{i=1}^{n_{h}} {(y_{h_{i}}-\overline{y}_{h})^2}. \end{equation}

The paired jackknife replicate variance estimator assigns one replicate \(h=1,…,H\) to each stratum, so that the number of replicates equals \(H\). In NAEP, the replicates correspond generally to pairs and triplets (with the latter only being used if there are an odd number of sample units within a particular primary stratum generating replicate strata). For pairs, the process of generating replicates can be viewed as taking a simple random sample \(J\) of size \(\frac{n_{h}}{2}\) within the replicate stratum, and assigning an increased weight to the sampled elements, and a decreased weight to the unsampled elements. In certain applications, the increased weight is double the full sample weight, while the decreased weight is in fact equal to zero. In this simplified case, this assignment reduces to replacing \(\overline{y}_{h}\) with \(\overline{y}_{h}(J)\), the latter being the sample mean of the sampled \(\frac{n_{h}}{2}\) units. Then the replicate estimator corresponding to stratum \(r\) is

\begin{equation} \hat{\overline{Y}}(r)=\sum_{h \ne r}^{H} W_{h} \overline{y}_h + W_r \overline{y}_h(J). \end{equation}

The \(r\)-th term in the sum of squares for \(\nu_{j} \left( \hat{\overline{Y}}\right)\) is thus

\begin{equation} \left( \hat{\overline{Y}}(r)- \hat{\overline{Y}} \right)^2 = W_r^2 \left( \overline{y}_r(J)- \overline{y}_r \right)^2. \end{equation}

In stratified random sampling, when a sample of size \(\frac{n_r}{2}\) is drawn without replacement from a population of size \(n_r\), the sampling variance is

\begin{equation} \begin{aligned} E \left( \overline{y}_{r’}(J)-\overline{y}_r \right)^2 = \frac {1} {\frac{n_r}{2} } \frac{ n_r - \frac{n_r}{2}} {n_r} \frac {1}{n_r-1} \sum_{i=1}^{n_r} \left( y_{r_{i}} - \overline{y}_r \right)^2 \\ = \frac {1} {n_r \left( n_r-1 \right) } \sum_{i=1}^{n_r} \left( y_{r_{i}} - \overline{y}_r \right)^2 = \frac {s^2_r}{n_r}. \end{aligned} \end{equation}

See for example Cochran (1977)​, Theorem 5.3, using \(n_r\), as the “population size,” \(\frac{n_r}{2}\)as the “sample size,” and \(s^2_r\) as the “population variance” in the given formula. Thus,

\begin{equation} E \left\{ W_r^2 \left( \overline{y}_{r}(J)- \overline{y}_r \right)^2 \right\} = W_r^2 \frac{s_r^2}{n_r}. \end{equation}

Taking the expectation over all of these stratified samples of size \(\frac{n_r}{2}\), it is found that

\begin{equation} E \left( \nu_j \left( \hat{\overline{Y}} \right) \right) =\nu \left( \hat{\overline{Y}}\right). \end{equation}

In this sense, the jackknife variance estimator "gives back" the sample variance estimator for means and totals as desired under the theory.

In cases where, rather than doubling the weight of one half of one variance stratum and assigning a zero weight to the other, the weight of one unit is multiplied by a replicate factor of \((1+\delta)\), while the other is multiplied by \((1-\delta)\), the result is that

\begin{equation} E \left( \hat{\overline{y}}(r)- \hat{\overline{y}} \right)^2 = W^2_r \delta^2 \frac{s^2_r}{n_r}. \end{equation}

In this way, by setting \(\delta\) equal to the square root of the finite population correction factor, the jackknife variance estimator is able to incorporate a finite population correction factor into the variance estimator.

In practice, variance strata are also grouped to make sure that the number of replicates is not too large (the total number of variance strata is usually 62 for NAEP). The randomization from the original sample distribution guarantees that the sum of squares contributed by each replicate will be close to the target expected value.

For triples, the replicate factors are perturbed to something other than 1.0 for two different replicate factors, rather than just one as in the case of pairs. Again in the simple case where replicate factors that are less than 1 are all set to 0, the replicate weight factors are calculated as follows.

For unit \(i\) in variance stratum \(r\)

\begin{equation} w_i(r) = \left\{\begin{array}{lll} 1.5w_i & i= \text{variance unit 1}\\ 1.5w_i & i= \text{variance unit 2}\\ 0 & i= \text{variance unit 3} \end{array}\right. \end{equation}

where weight \(w_i\) is the full sample base weight.

Furthermore, for \(r'=r+31\) \(mod\) \(62\)​

\begin{equation} w_i(r') = \left\{ \begin{array}{llll} 1.5w_i & i= \text{variance unit 1}\\ 0 & i= \text{variance unit 2}\\ 1.5w_i & i= \text{variance unit 3} \end{array}\right. \end{equation}

And for all other values \(r^*\), other than \(r\) and \(r'\), \(w_i \left(r^*\right)=1\).

In the case of stratified random sampling, this formula reduces to replacing \(\overline{y}_r\) with \(\overline{y}_r(J)\) for replicate \(r\), where \(\overline{y}_r(J)\) is the sample mean from a "\(2/3\)" sample of \(\frac{2n_r}{3}\) units from the \(n_r\) sample units in the replicate stratum, and replacing \(\overline{y}_r\) with \(\overline{y}_{r'}(J)\) for replicate \(r'\), where \(\overline{y}_{r'}(J)\) is the sample mean from another overlapping "\(2/3\)" sample of \(\frac{2n_r}{3}\) units from the \(n_r\) sample units in the replicate stratum.

The \(r\)-th and \(r'\)-th replicates can be written as

\begin{equation} \hat{\overline{Y}}(r)=\sum_{h \ne r}^{H} W_{h} \overline{y}_h + W_r \overline{y}_r(J), \end{equation}

\begin{equation} \hat{\overline{Y}}(r')=\sum_{h \ne r}^{H} W_{h} \overline{y}_h + W_r \overline{y}_{r'}(J). \end{equation}

From these formulas, expressions for the \(r\)-th and \(r'\)-th components of the jackknife variance estimator are obtained (ignoring other sums of squares from other grouped components attached to those replicates):

\begin{equation} \left( \hat{\overline{Y}}(r)- \hat{\overline{Y}}\right)^2= W^2_r \left( \overline{y}_r(J)- \overline{y}_{r}\right)^2, \end{equation}

\begin{equation} \left( \hat{\overline{Y}}(r’)- \hat{\overline{Y}}\right)^2= W^2_r \left( \overline{y}_{r’}(J)- \overline{y}_{r}\right)^2. \end{equation}

These sums of squares have expectations as follows, using the general formula for sampling variances:

\begin{equation} \begin{aligned} E\left( \overline{y}_r(J)- \overline{Y}_r\right)^2= \frac {1}{\frac{2n_r}{3}} \frac {n_r- \frac{2n_r}{3} }{n_r} \frac {1}{n_r-1} \sum_{i=1}^{n_r} \left( y_{r_{i}} - \overline{y}_r \right)^2 \\ =\frac{1}{2n_r \left( n_r-1\right)} \sum_{i=1}^{n_r} \left( y_{r_{i}} - \overline{y}_r \right)^2 \\ =\frac {s^2_r}{2n_r}, \end{aligned} \end{equation}

\begin{equation} \begin{aligned} E\left( \overline{y}_{r’}(J)- \overline{Y}_r\right)^2= \frac {1}{\frac{2n_r}{3}} \frac {n_r- \frac{2n_r}{3} }{n_r} \frac {1}{n_r-1} \sum_{i=1}^{n_r} \left( y_{r_{i}} - \overline{y}_r \right)^2 \\ =\frac{1}{2n_r \left( n_r-1\right)} \sum_{i=1}^{n_r} \left( y_{r_{i}} - \overline{y}_r \right)^2 \\ =\frac {s^2_r}{2n_r}. \end{aligned} \end{equation}

Thus,

\begin{equation} \begin{aligned} E \left\{ W_r^2 \left( \overline{y}_r(J)- \overline{y}_r \right)^2 + W_r^2 \left( \overline{y}_{r’}(J)- \overline{y}_r \right)^2 \right\} \\ = W_r^2 \left( \frac {s^2_r}{2n_r} + \frac {s^2_r}{2n_r} \right) \\ = W_r^2 \frac {s^2_r}{n_r}, \end{aligned} \end{equation}

as desired again.

Printer-friendly Version