• Introduction
• Program Outcomes: Measuring the Impact of Vocational Education
• Program Characteristics: Program Attributes and Participant Demographics
• Conclusion
• References

Accountability has become the watchword for education policy and will continue to dominate public discussion in the coming years. Vocational education has, in many respects, helped lead this new emphasis on results. The Carl D. Perkins Vocational and Technical Education Act of 1998, building on requirements for performance indicators and standards in earlier vocational education legislation, directed the states to develop systems of accountability incorporating four core indicators of student performance:

• student achievement, academic as well as technical;
• completion of coherent programs of technical and academic study and attainment of a high school diploma and postsecondary degrees and certificates;
• successful transition from secondary to postsecondary education and from education to the labor force; and
• equity, with respect to gender as well as special needs.

In principle, few would take issue with this attention to results. In the final analysis, what other justification can there be for sustaining particular programs of study and student support services if they do not produce observable gains in desired student outcomes? In practice, however, obtaining credible evidence of improved student performance has proven quite difficult. For a variety of reasons, by no means unique to vocational education, policy deliberations have lacked solid evidence on the impact of particular initiatives and the public dollars invested in them. Providing better information on what happens to students who participate in secondary and postsecondary vocational education is perhaps the greatest challenge facing the future development of national data to inform policy focused on this constellation of programs and services.

How can the National Center for Education Statistics (NCES) help address this challenge as it maintains and improves the Career/Technical Education Statistics (CTES) system, the primary system for generating credible national estimates on key aspects of secondary and postsecondary vocational education? This commentary suggests a possible agenda for data development over the next 5 to 10 years. It examines two major categories of information: (1) program outcomes, including student achievement and participation in further education and the labor force; and (2) program characteristics, including particular attributes of programs and services, as well as the demographics of participating students.

During the evolution of CTES over the past 15 years, much has been accomplished. Today, as a result of well-designed transcript studies in both secondary and postsecondary education, we have much better information on who participates in the vocational curriculum, the kinds of courses and programs in which they enroll, how many credits or Carnegie units students accumulate in particular subjects, and the types of grades earned. Through longitudinal studies, we are better able to define and trace the different pathways students pursue through high school and postsecondary education as well as examine interactions between school and work. The National Postsecondary Student Aid Study (NPSAS) and its companion Beginning Postsecondary Students Longitudinal Study (BPS) tell us with high levels of precision who participates in postsecondary programs, what patterns of persistence and completion they follow, and how these estimates differ among different types of postsecondary education. With regard to both precision and detail, we know a great deal more about vocational education today than we did in the early 1980s, when the Vocational Education Data System (VEDS) came to an end.

Nevertheless, there is still much that we do not know. Most importantly, we do not yet have good information on what students learn from participating in vocational education, whether in technical courses alone or in combination with academics or other forms of learning such as youth apprenticeships or cooperative education. Additionally, while longitudinal studies have furnished better data on labor market participation by students taking vocational courses, one of the key labor market outcomes-earnings, immediately before and after participation in vocational education and over time-is not yet accurately or consistently measured. Finally, for a large and growing number of older adults using the postsecondary vocational education system for short-term skill upgrading, we know virtually nothing about learning gains or impacts on employment and earnings. In short, today we have much better information on who participates in vocational education but still know very little about what is accomplished as a result.

Better understanding the results of vocational education will require attention to improving the measurement of at least two kinds of outcomes: (1) learning gains and (2) labor force effects. On both fronts, there are challenging conceptual issues that need attention, in addition to definitional and methodological concerns.

Traditionally, vocational education was expected to impart to students occupationally specific knowledge and skills that would enable them to secure entry-level employment, especially for those students entering the labor force immediately after high school. As vocational instruction assumed greater importance in community colleges, private proprietary schools, and other kinds of less-than-4-year postsecondary institutions, the technical rigor of vocational instruction increased. Nevertheless, vocational education was still intended primarily as preparation for work requiring less than a baccalaureate degree.

Although federal legislation still defines vocational education as having a subbaccalaureate focus, this limitation has gradually been disappearing in actual practice. With the increased necessity of having some postsecondary education to sustain employability and with the growing earnings gap between people who have a baccalaureate degree and lesser levels of postsecondary attainment, there is a growing consensus that vocational education should contribute to students' pursuit of the full range of postsecondary and employment options, not just entry-level or subbaccalaureate opportunities.

With this broadening of mission has also come an expansion of learning objectives. Thus, for the first time in its 80-year history, federal vocational education legislation in 1998 explicitly stressed that vocational education should contribute to students' mastery of academics, as well as technical knowledge and skills. Moreover, several of the fastest growing programmatic innovations in vocational education-career academies and tech-prep, for example-aim to offer students a comprehensive program of closely linked academic and technical studies, often spanning 2 to 4 years of secondary and postsecondary instruction.

That vocational education should help students learn and apply academic knowledge and skills seems beyond dispute. However, defining precisely what this means, determining how best to measure it, and assessing whether this expanded policy objective is being met are quite problematic.

Furthermore, it is not just assessing academic achievement that is difficult. Ironically, despite its longstanding emphasis on occupationally related knowledge and skills, vocational education lacks any widely accepted, rigorously validated assessments of students' technical achievement. Additionally, recent initiatives to expand work-related learning to include more generic proficiencies such as problem solving, understanding of systems, and the ability to work in teams have yet to produce credible assessment instruments. What, then, are the prospects that national data will be able to speak to these aims in the near future?

Academic achievement. At the secondary level, participation in vocational education occurs mostly during grades 11 and 12 (although career academies and some tech-prep programs span 3 or 4 years of high school). Thus, if vocational education is to contribute to students' academic achievement, these effects are most likely to occur during the last 2 years of high school and are probably best measured by an assessment administered near the end of the 12th grade.

Although few states assess students' academic performance in the 12th grade, the National Assessment of Educational Progress (NAEP) does test a national sample of approximately 11,000 12th-graders. Future assessments of 12th-grade mathematics and science are slated for 2004 and 2008, with assessments of reading and writing scheduled for 2002, 2006, and 2010.

With respect to content, the NAEP assessments are reasonable indications of the kinds of academic knowledge and skills vocational education might be expected to reinforce. The mathematics assessment, for example, measures students' conceptual knowledge, procedural understanding, and problem-solving skills, as well as their abilities to reason, communicate, and make connections mathematically. However, as a primary tool for determining the contribution of vocational education to academic achievement, NAEP in its current design has two, possibly three, shortcomings.

First, NAEP is not designed to produce test scores for individual students; rather, it yields estimates for subpopulations. Consequently, analyzing relationships between particular patterns of vocational and academic coursetaking (using data from the transcript studies that accompany NAEP) and levels of academic achievement is methodologically challenging. This problem would be less daunting were it not for a second limitation of NAEP, sample size.

The absence of NAEP scores for individual students would be less problematic if it were possible to create usable subpopulations of students participating in secondary vocational education. However, the size of the national sample of 12th-graders (between 10,000 and 11,000 students) hampers most detailed analysis of patterns of participation in vocational education. Probably fewer than 2,000 of these 12th-graders have taken three or more units of vocational education in a specific program area. Even fewer have been part of a career academy or enrolled in tech-prep (conditions that presently cannot even be determined from NAEP data), making it very difficult to examine the achievement effects for students participating in such relatively new and emerging strategies for strengthening vocational instruction.

Finally, as good an indicator of academic achievement as NAEP is, it still may not be an adequate instrument for analyzing the contribution of vocational education to academic achievement. For one thing, it may not be calibrated finely enough to detect the influence of two or three units of vocational education on academic achievement. For another, it may not measure one of the most likely impacts of vocational education on academic achievement, namely, students' ability to apply selected academic concepts and skills to the kinds of problems or situations typically encountered in the world of work.

In sum, producing better measures in national surveys of vocational education's contribution to academic achievement will require at least two important actions. First, it will probably be necessary to improve the assessment instruments to better capture the potential impact of vocational education on academic achievement. Second, increasing the sample size of 12th-graders, either overall or by oversampling students who concentrate in vocational education generally or in particular programs, would make estimation easier and more precise.

Technical achievement. Regardless of NAEP's shortcomings, it nevertheless provides a rigorous basic foundation on which one might build better measures of vocational education's impact on academic achievement. Unfortunately, there is presently no comparable footing on which to construct an assessment of what vocational education contributes to students' attainment of technical knowledge and skills. On this score, there are several conceptual and methodological impediments.

Conceptually, there is a basic unanswered question: what to measure? Traditionally, formal assessment in vocational education has focused on students' mastery of occupationally specific competencies, as evidenced through either paper-and-pencil exams or actual performance demonstrations. The specification of competencies can be quite specific and formal as, for example, in the case of the knowledge and skills required of airframe and power mechanics as overseen by the Federal Aviation Administration. Alternatively, the definition of competencies and the expected levels of performance can be left entirely to the discretion of local instructors and their own development of particular assessment practices. As a result, what is measured varies widely, with little or no standardization or quality control, across vocational programs and the many secondary and postsecondary institutions offering instruction.

To help clarify the knowledge and skills expected of students pursuing occupational education and training and to develop a manageable system of voluntary industry skill standards, assessment, and certification, Congress in 1994 created the National Skill Standards Board (NSSB). NSSB is building a system organized around 15 major industries. Work is nearing completion in two of these sectors, manufacturing and wholesale/retail, and is well under way in about six others. A fully developed system, however, is still many years away. It is possible, however, that as NSSB's work on individual sectors is finished, NSSB standards and assessment procedures could be incorporated into such national education data systems as NAEP, NPSAS, and the Education Longitudinal Study of 2002 (ELS:2002).

NSSB, by organizing its system around 15 industry sectors rather than the hundreds of occupationally specific job classifications that have been the focus of most other assessment and certification initiatives, may greatly simplify assessment of technical knowledge and skills. Nevertheless, direct assessment of student achievement in 15 different industries is likely to still be quite difficult to do in various national education surveys. Even if valid assessment and certification procedures are developed in each of these sectors, it is by no means certain that they will rely mainly on the paper-and-pencil examinations that are the typical instruments of large-scale national assessments. Moreover, the mechanics of determining how to administer 15 different kinds of assessments would certainly be complicated.

Consequently, to include measures of technical achievement in national surveys it may be necessary to rely on more indirect strategies. If, for example, NSSB succeeds in designing solid assessments and if these become widely employed in secondary and postsecondary institutions, national surveys could concentrate on capturing student scores or evidence of competency-based program completion from administrative records. Similarly, national surveys might seek to retrieve information on certification of students in industries or occupations (e.g., nursing, automotive technicians, and aviation occupations) where certification through industry-developed systems or state certification requirements is becoming more rigorous and uniform throughout the country.

Finally, assessment of students' mastery of technical knowledge and skills will need to pay more attention to more generic outcomes such as the ability to manage resources, use information, comprehend systems, employ interpersonal skills, and understand technological principles and applications. Such proficiencies-representing the consensus of the Secretary's Commission on Achieving Necessary Skills (SCANS) on what is required to succeed in high-performance workplaces-are increasingly the focus of vocational and technical education in the nation's high schools and colleges. Nevertheless, there are not yet good instruments for assessing these new learning objectives.

As one would expect, a key aim of vocational education is improving students' prospects for sustained, rewarding employment. The current national surveys provide reasonably good data on such labor market outcomes as time to employment, type of employment (occupation or industry), and amount of time employed or unemployed. However, data on one essential indicator-earnings-leave much to be desired. All of the national surveys rely on respondents to self-report information on wages and salaries, and the resulting measures suffer with respect to accuracy, comparability, and consistency over time.

More direct and precise measures of earnings are difficult to obtain. One tempting strategy is to merge survey data with wage information from W-2 forms submitted to the Internal Revenue Service, but legitimate concerns about protecting the integrity of the income tax system have so far ruled out this option. An alternative being used by an increasing number of states interested in tracking employment and earnings of students is to merge student record information with data on employment and earnings from their unemployment insurance systems, which maintain accurate records of quarterly earnings by most workers.*

While merging national survey data with state unemployment insurance records would be logistically challenging, this approach is becoming more practical as more and more states gain experience with the technique. Sampling strategies that concentrated on particular states, for example, might yield much better earnings data. Alternatively, asking respondents to supplement self-reported earnings with paper documentation, such as a pay stub, might help improve the quality of this important measure.

Finally, it should be noted that for a rapidly growing population of participants in vocational education-older adults returning for skill upgrading and retraining-it is as important to have data on earnings immediately prior to enrollment as well as after completion. Measuring "value added" is the preferred methodology for understanding benefits to individuals who already have extensive labor force experience, and this cannot be done only with earnings information following program participation. Moreover, because this kind of participation is of much shorter duration than the degree and certificate programs typically pursued by younger students, more precise estimates of pre- and post-earnings are essential if the effects of vocational instruction are to be accurately evaluated.

Although CTES has greatly improved information on who is participating in vocational education and on what they are taking, there are some aspects of program definition and the attributes of participants that need attention as the design of future surveys proceeds. The delivery of vocational education is quite varied across the country, especially with the policy emphasis on new forms of academic and vocational integration and better articulation between secondary and postsecondary offerings. Similarly, participation is very diverse, with many different subpopulations of students taking vocational courses, often for very different reasons. Current surveys could do a better job of capturing these distinctions.

With respect to better distinguishing the types of vocational program offerings, surveys must make a better effort to identify tech-prep programs and career academies. Tech-prep programs-comprehensive programs of academic and technical study spanning the last 2 years of high school and the first 2 years of postsecondary education-have been a priority of federal policy since the mid-1980s, but national surveys do not distinguish these offerings from other types of vocational education.

Similarly, career academies are one of the fastest growing new forms of integrated academic and technical instruction in high schools throughout the country. A recent rigorous evaluation of career academies by the Manpower Demonstration Research Corporation (Kemple and Snipes 2000), employing experimental design, concluded that these programs are having significant benefits, especially for at-risk students, with respect to attendance, grades, and high school completion. National Career/Technical Education Statistics should track participation in career academies and monitor outcomes associated with participation. Better determination of students' participation in various forms of work-based learning-such as youth apprenticeships, cooperative education, and internships-would also be helpful.

Regarding the types of students participating in vocational education, better information on certain subpopulations would be helpful. Congress has long been concerned about benefits to students with a wide range of special needs. Some of these populations are relatively small and, therefore, often are not well represented in the samples used for national surveys. More attention to oversampling certain subpopulations or other strategies for improving representation would be helpful. Among the subpopulations to be considered for more substantial inclusion are students enrolled in special education and students with limited English proficiency.

During the past 2 decades, NCES has achieved considerable improvement in the accuracy, comparability, and consistency of Career/Technical Education Statistics and on vocational education's relationship to the larger secondary and postsecondary enterprise in which it operates. Most significantly, basic questions about who participates in vocational education and what they take can be answered with confidence, and good trend data over the past 15 to 20 years are now taken for granted.

The challenge for the next decade is to better understand what participation in vocational education means. How does it contribute to students' mastery of academic and technical knowledge? What effects does it have on opportunities to pursue further education and participate successfully in the labor force? On this score, much remains to be done.

Because CTES is an "integrated" data system, relying on a variety of different surveys, none of which has as its primary purpose collecting information about vocational education, it is difficult to systematically modify and maintain a comprehensive, up-to-date plan for collecting national Career/Technical Education Statistics. As CTES approaches its 20th year of operation, it is perhaps time for a major concentrated review. The top priority for such an examination should be ambitious, but realistic efforts to define and measure the learning gains and employment outcomes of vocational education participants in national surveys. Successfully addressing this objective will help ensure that national policy on vocational education will be well informed in the years ahead.

Kemple, J.K., and Snipes, J.C. (2000). Career Academies: Impacts on Students' Engagement and Performance in High School. New York: Manpower Demonstration Research Corporation.