SASS Principal Follow-Up Survey (PFS)

5. Data Quality and Comparability

Sampling Error

Because the PFS is a subsample of the SASS or NTPS principal sample and the PFS nonresponse rate was extremely low, no new replicate weights were calculated for the 2008–09 PFS and the 2012–13 PFS. Instead, the 2007–08 SASS public, private, and BIE principal replicate weights and the 2011–12 SASS public and private replicate weights should be used when analyzing the public, private, and BIE principal status data files. For more information on the SASS replicate weights, please see the Documentation for the 2011–12 Schools and Staffing Survey ( Cox et al ., 2016).

 

Top

Nonsampling Error

Coverage error. There is a potential for bias to be introduced into PFS because the PFS frame only includes principals who responded to SASS or NTPS.

Unit nonresponse.The unit response rate is the rate at which the sampled units respond by completing the questionnaire. For PFS, any questionnaire with question 1 answered was considered a complete interview. The weighted unit response rates were produced by dividing the base-weighted number of respondents who completed the questionnaires by the base-weighted number of eligible sampled cases. The base weight for each sampled unit is the inverse of the probability of selection.

The overall response rate represents the response rate to the PFS taking into consideration each stage of data collection. To be eligible for the PFS, it was necessary for the principal to have responded to the SASS or NTPS principal questionnaire. Thus, the overall unit response rate is the product of the sampled PFS unit response rate multiplied by the SASS or NTPS principal questionnaire unit response rate. The product of these gives the cumulative nonresponse. The overall unit response rate was calculated using base-weights.

For the 2008–09 PFS, the overall weighted unit response rate for public and BIE school principals was almost 80 percent; the overall weighted unit response rate for private school principals was lower (70 percent). For the 2012–13 PFS, the overall weighted unit response rate was 73 percent for traditional public school principals, 70 percent for public charter school principals, and 65 percent for private school principals. For the 2016–17 PFS, the overall weighted unit response rate was 69 percent for traditional public school principals and 67 percent for public charter school principals.

 

Item nonresponse. The item response rate indicates the percentage of respondents who answered a given survey question or item. The weighted item response rate is calculated by dividing the weighted number of respondents who provided an answer to an item by the weighted number of respondents who were eligible to answer that item.

For the PFS, only a response to the principal’s current occupational status is needed for an interview to be considered complete. Therefore, the item response rate of question 1 is the same as the unit response rate. For the 2008–09 PFS, the weighted unit response rate for public and BIE school principals was 99 percent; the weighted unit response rate for private school principals was slightly lower (97 percent). For the 2012–13 PFS, the weighted unit response rate was nearly 100 percent for traditional public school principals, 99 percent for public charter school principals, and 96 percent for private school principals. For the 2016–17 PFS, the item response rate for the PFS item was 96 percent for traditional public school principals, and 93 percent for public charter school principals. Since the item response rate was above 85 percent, a nonresponse bias analysis for the item was not necessary. See table PFS-1 for details on the response rates.

Measurement error. Re-interviews were conducted in the validation study for the purpose of measuring response variance in the 2012–13 PFS. In general, more variation in response was found than was expected. This was particularly troublesome for the stayer category, where the expectation was that the principal’s current status would be fairly unambiguous. Anecdotal evidence gathered seems to point to confusion over the term “principal” and precisely who it may apply to at a particular school. The ability to draw definitive conclusions was limited by the small sample size captured by the validation study and the inability to reach a substantial minority of the original validation study sample. In particular, subpopulations could not be compared, so it is not clear under what circumstances it may be acceptable to accept proxy responses.

Data Comparability

Care must be taken in estimating change over time with PFS data files, because some of the measured change may not be attributable to a change in the educational system, but due to changes in the sampling frame, or other changes.  For example, 2008–09 PFS release reports have included information on principals in public, public charter, private, and BIE schools during the 2007–08 SASS, but only public, public charter, and private school principals are included for the 2012–13 PFS reporting due to a sampling frame change.

Since the 2015–16 NTPS only included public schools, private school principals were not included in the 2016–17 PFS. For further information on data elements, see Documentation for the 2008–09 Principal Follow-up Survey (Battle et al., 2010) and Documentation for the 2012–13 Principal Follow-up Survey (Cox and Cox, 2015).

Top