Table of Contents  |  Search Technical Documentation  |  References

NAEP Instruments → Cognitive Items and Instruments → Bundling of the Student Booklets → Improving the Vertical Bundle Map

NAEP Technical DocumentationImproving the Vertical Bundle Map

There are a number of problems that can surface in what may appear to be a perfectly serviceable bundle map. Changing from a horizontal plan to a vertical plan addresses one of these problems—badly balanced counts of pairs of booklets. Vertical maps can also display unwanted characteristics. Here are four such problems:

Repetitive Bundle Bias

It sometimes happens that the vast majority of bundle shipments calls for the same multiple of bundles. Suppose that almost all testing sessions call for more than one bundle but less than two. This means that two bundles will be routinely shipped together. If, in addition, the total number of bundles is even, all these shipments will contain an odd-numbered bundle followed by an even-numbered bundle. The result will be that booklets occurring toward the end of an even-numbered bundle will be less likely to be used. If these booklets are not representative of the full spiral, there will be bias in the final counts. This problem can occur with any multiple of routine shipments of three bundles when the total number of bundles is a multiple of three or more.

Subject Runs

A well-designed bundle map is intended to provide a good distribution of the various booklets, both globally and locally. Such a map will provide a good distribution of the various booklets in each session administered, regardless of the size of that session. When there are too many consecutive booklets of the same subject in a given bundle, it is possible that the session receiving such a bundle will not offer a balanced look at the various intended subjects in the assessment. The effect will be repeated each time this bundle is used, multiplying the resulting imbalance. This is undesirable psychometrically, so runs of the same subject are avoided in NAEP bundle maps.

Within-Session Duplicates

When a session includes an excessive number of duplicate booklets, there is a corresponding loss of domain coverage in the session. As noted above in the discussion of subject runs, it is psychometrically desirable to have a representative distribution of the various booklets both globally and locally in the bundle map so the best possible measure of subject abilities in the smallest unit of assessment—the session—is attained.

Inadequate Spiral Distribution

Working through a bundle map horizontally (which was the way the booklets were used in the field), one would like to see the distribution defined by the spiral in consecutive sets. For example, if a spiral consists of 100 booklets, one wants to see the subject distribution represented by those booklets in the course of “reading” through each successive set of 100 booklets in the bundle map. This gives some protection against a biased distribution in some groups of sessions (such as a full school district or even an entire state).

To address these problems, a number of techniques have been developed to improve the bundle map constructed from a vertical design, which is the type of design generally used in NAEP:

• bundle permutation,
• map doubling,
• column permutation, and
• horizontaling.

While new approaches to bundling are constantly under study, any or all of the techniques may be used to post-process a “raw” vertical design in an effort to free it from any or all of the above-listed problems. It will be useful to refer to a model bundle map during this discussion, so let’s repeat the 8 x 4 x 8 design from the previous section that introduced bundling.

Of course an actual NAEP bundle map would usually be much larger than this, with hundreds of rows and 14 to 16 columns. Any problems or flaws in such a map may not be obvious to the unaided eye. All potential NAEP bundle maps are examined by a number of programs that have been developed for that purpose. Even without the assistance of this body of inspection software, however, the last three positions of every even-numbered bundle contain odd-numbered booklets (as shown in the figure above). If two bundles are shipped (odd, followed by even) routinely, there will be a large bias in favor of the even-numbered booklets because the odd-numbered booklets fall at the end of a double bundle—an example of Repetitive Bundle Bias, discussed above.

In recent years, NAEP spirals have become quite complex, involving several subjects in combinations meant to satisfy several different sample sizes. It becomes correspondingly difficult to design a bundle map that meets these needs. Unforeseen events may affect the intended distributions. No events have seriously impacted NAEP results; the bundle maps used have proven to be a sufficiently robust method for distributing booklets to students to allow a data collection effort that satisfies NAEP analysis needs.

Bundle Permutation

Bundle permutation is a systematic permutation of the bundles in a map to minimize the number of within-session duplicates. It is not feasible to try all n! (n = number of bundles) permutations, so instead a number is selected which is relatively prime with the number of bundles; this is used as a stride length beginning with the first bundle. In the sample below, the number 5 proved to be the most successful at eliminating duplicates. The resulting bundle sequence is 1, 6 (1 + 5), 3 (6 + 5, mod 8), 8, 5, 2, 7, and 4. This permutation, however, did not correct the Repetitive Bundle Bias, which brings us to map doubling.

Map Doubling

Whenever the resulting number of bundles is not excessive, the map can simply be doubled using the opposite pairing in the replicate. If one wanted to take advantage of the bundle permutation, the resulting map after doubling would be the following sequence of bundles: 1, 6, 3, 8, 5, 2, 7, 4, 6, 1, 8, 3, 2, 5, 4, and 7. Notice that when the first pair of bundles (1,6) is repeated, the sequence is reversed (6,1). By reversing the sequence of each pair in the second half of the double map, the bias that might occur from the positioning of the bundles during shipping of pairs of bundles (a common practice) is canceled.

Column Permutation

Another way to deal with the bias noted in the sample map below is to permute columns. If columns 1 and 4 are switched, the result turns out to have no booklet usage bias, even with regular shipping of pairs of bundles. It is tempting to make use of this option since it involves only half as many bundles as the map doubling strategy. In practice, it may be much harder to discover a column permutation that not only solves the problem, but also does not introduce any new ones. As a result, map doubling is more likely to be used routinely when there are an even number of bundles.

Horizontaling the Bundle Map

A recently developed tool for improving a vertical bundle map is permuting booklets within a column in order to introduce some of the benefits of a horizontal design. This can help with all of the problems noted above. The method involves reading through the map, one bundle at a time, and trying to match the subject alternation found in the spiral cycle. When the “wrong” subject is found, an attempt is made to swap this booklet with one from the “correct" subject among the booklets in the same column lower down in the map. In many maps, this process greatly improves the distribution parameters.

Printer-friendly Version