Table of Contents  |  Search Technical Documentation  |  References

NAEP Instruments → Cognitive Items and Instruments → Bundling of the Student Booklets

NAEP Technical DocumentationBundling of the Student Booklets

			Example of a Bundle Map Reducing the Size of Bundle Maps Improving the Vertical Bundle Map

Until 2003, NAEP used a horizontal scheme as the booklet bundling method. The spiral cycle, which is the minimum number of booklets required to satisfy the sample proportions, is repeated over and over along horizontal lines, with the line length (bundle length) chosen so that each booklet in the spiral cycle occurs once in each bundle position across all bundles. This is achieved by consecutively repeating the spiral cycle until each booklet has appeared in each bundle position; the minimum set of bundles required for that is called the "bundle map." This can only be done if the spiral length and the bundle length are relatively prime; in other words, the two lengths do not share a common factor.

Shown below is an example of eight booklets in a spiral with a bundle length of three. Each of the eight booklets appears once in each of the three positions across the eight bundles.

This bundle map, in which each number refers to a specific booklet, shows the ordering of booklets in the eight bundles. The need for the bundle length to be relatively prime to the spiral length becomes clear if one tries to distribute these eight booklets in bundles of, for example, four:

This arrangement leads to a serious positional imbalance. Booklets 1 and 5, for example, always occur in the first position of a bundle, giving them a much better chance of being distributed in the field than, say, Booklets 4 and 8, which always occur in the last position. The relative shortage of Booklets 4 and 8 used in the field will call for larger samples to achieve the target sample sizes for the various booklets.

By choosing bundle lengths relatively prime to the spiral lengths, the traditional NAEP bundling scheme achieves balanced booklet usage, so that each booklet in the spiral cycle has an equal chance of occurring in each bundle position and hence an equal probability of being used in the field. The requirement of the bundle length being relatively prime to the spiral length, however, limits the number of choices for bundle lengths, and usually results in a large number of bundles, which is unwieldy and inconvenient. Another disadvantage of the horizontal scheme is that it does not control for within-bundle booklet pairings. Failure to control booklet pairing will result in some booklet pairs being more common in a session than other booklet pairs. In the two examples shown above, Booklets 1 and 2 are expected to occur together in the same session much more frequently than Booklets 1 and 5, simply because Booklet 1 is always followed by Booklet 2 (if it is followed by any booklet) in a bundle. This imbalance of booklet pairs could jeopardize the assumption that all items are being presented to the same population of examinees.

In 2003, vertical bundling was implemented and continues to be used in current assessments for most bundle maps. In vertical bundling, the spiral cycle is repeated vertically across all the bundles. Shown below is an example of a vertical design with eight booklets in a spiral using a bundle length of four.

The vertical bundle scheme is created based on a technique used in the construction of a cyclic Youden Rectangle. A Youden rectangle can be seen as a special case of a BIB in the sense that four treatments within blocks of a BIB are reordered so that both treatment positions and comparisons are balanced (Beall 1971). Applied to booklet bundling, with each booklet considered as a specific treatment and each bundle a unique block, a Youden design would have the potential for balancing both booklet position and within-bundle booklet pairing. It also is more flexible with respect to bundle length, since it potentially can be of any length rather than requiring a bundle length relatively prime to the spiral length.

While it is modeled after a cyclic Youden, the vertical bundling scheme does not usually have the complete balance a Youden has. For example, while booklets 1 and 2 appear together in the same bundle twice (in bundle 1 and bundle 2), booklets 1 and 4 appear in the same bundle only once (in bundle 1). In an actual Youden rectangle, within-bundle pairings would all appear equally often. Such balance is not possible in a design with 8 rows, 4 columns, with 8 booklets. Eight booklets can be paired in exactly 28 ways; the 4 booklets in a bundle (row) give 6 pairings, yielding 6 times 8 bundles, or 48 pairings in the entire design. Forty-eight is not an integral multiple of 28, so the 8 x 4 x 8 design cannot be balanced exactly. Nevertheless, an adequate job for “partial” balance was done here: All booklet pairs appear in a bundle either once or twice (the very best that can be done with this design); all booklets are equally represented in each bundle position.

As can been seen from the above examples, for a spiral that involves eight different booklets, the bundle length four can be used for the vertical scheme, but it is an invalid length for the horizontal scheme because four and eight share the common factors two and four. Furthermore, with even numbered bundle length, the vertical bundle scheme can reduce the required number of bundles to one-half or even one-quarter of that required by a horizontal scheme, which is beneficial since a lower number of different bundles for a specific spiral facilitates bundle storage and maintenance.

Printer-friendly Version