School-based programs designed to provide students with the credentials and skills needed to make successful transitions to post-secondary education and careers.
Blueprints: Promising
Crime Solutions: Effective
OJJDP Model Programs: Effective
Social Programs that Work:Top Tier
What Works Clearinghouse: Meets Standards Without Reservations - Positive Effect
Erin Fender
College and Career Academy Support Network
University of California
Berkeley Graduate School of Education
Berkeley, CA
Email: efender@berkeley.edu
(707) 591-6375
http://casn.berkeley.edu/index.php
College and Career Academy Support Network
Career Academies are school-based programs that seek to reduce dropout rates and improve school performance and career readiness among high-school youth. A Career Academy (CA) is organized as a school-within-a-school, where students work in "small learning communities." Each small learning community involves a small number of students working with the same group of teachers for three or four years of high school with the aim being to create a more personalized and supportive learning environment for students. CAs offer students a combination of academic and career-technical curricula and use a career theme to integrate the two. In an effort to build connections between school and work and to provide students with a range of career development and work-based learning opportunities, CAs establish partnerships with local employers. To encourage post-graduate education they also build linkages to local colleges through curricular articulation, dual enrollment programs, and field trips to 2- and 4-year institutions.
A Career Academy is a personalized small learning community within a high school, comprised of a subset of students and teachers for a two-, three-, or four-year span. The goals of the Career Academy are to enhance students' engagement and performance in high school and provide them with the credentials and skills needed to make successful transitions to post-secondary education and, eventually, a career. Students enter an academy in 9th or 10th grade through a voluntary process; they must apply and be accepted, with parental knowledge and support. As academies function within the larger high school, they require a great deal of administrator and counselor support. Academies vary in size, usually including one to three sections of students at each grade level, or 100-300 students in all. Academy classes are generally blocked back-to-back in the daily schedule, and students attend them as a group. Students are able to complete academy requirements within the regular school day, with the exception of work internships and possible college classes.
An academy involves teachers from different subject areas working together as a team. This team manages the program, with one member usually serving as the coordinator. Teams usually participate in professional development, particularly in implementing the key features of the learning model and gaining exposure to the career field. Team members have shared planning time and usually a daily common planning period. The joining of a group of students for several periods each day with teachers whom they have come to know well provides a family-like atmosphere and serves to nurture close student-teacher ties. Academy students may also participate in required and elective classes outside the academy, as well as other activities such as clubs and sports.
Students in a Career Academy have a mixture of career classes (usually one or two) and academic classes (usually three or four) at a time. The academic and some of the career-technical classes meet entrance requirements for four-year colleges and universities. They are linked to academic and industry standards and encourage high achievement. They show students how their subjects relate to each other and the career field.
The career classes develop knowledge in a given field. They are designed to expose students to the full range of careers in that field. Special projects require students to bring together academic skills across their subjects and apply these to community and work settings outside the school. Usually the junior and/or senior year includes work experience, a paid or unpaid work internship or community service assignment. During the senior year students are provided with college and career counseling, forming a post-graduate plan which may include college, a mixture of work and college, or full-time work. The majority of graduates do attend college at some level, most commonly at two-year institutions.
The academy career theme is selected locally, based on an industry that is healthy and can provide a cadre of partners interested in supporting the program. Employers from a group of companies in the selected field work as partners in the academy, serving on a steering committee (along with teachers, administrators, and often parents and students) that governs the program's development and operation. This committee helps to plan the various activities in which employee volunteers participate: as speakers at the school, informing students of the industry and career options; as field trip and job shadowing hosts at their companies; as individual mentors, career-related "big brothers and sisters;" as work internship supervisors during the summer following the junior year or part-time during the senior year; and as community service coordinators. The employer partners may also hire graduates. Postsecondary educational institutions are usually included as well, providing course articulation and concurrent enrollment options.
Primary Evidence Base for Certification
Study 1
Kemple et al. (2000, 2004, 2008a, 2008b) found at the 12th grade survey that:
8 Years After Students' Scheduled Graduation:
Primary Evidence Base for Certification
Of the three studies Blueprints has reviewed, one (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness). The study was conducted by independent evaluators.
Study 1
Kemple et al. (2000, 2004, 2008a, 2008b) used a large scale, multi-site, random assignment research design. The researchers studied nine CAs with a sample size of 1,764 students. Students in the sample were identified in the 8th or 9th grade and were followed through their senior year of high school and up until eight years post-graduation. Data utilized in the study consisted of survey information provided by both Academy and non-Academy students in the study sample, performance indicators obtained from school records and transcripts, and standardized test scores from a test the researchers administered to the sample of students.
Study 1
Kemple, J. J. (2004). Career Academies: Impacts on labor market outcomes and educational attainment. Manpower Demonstration Research Corporation (MDRC), www.mdrc.org.
Kemple, J. J., & Snipes, J. C. (2000). Career Academies: Impacts on students' engagement and performance in high school. Manpower Demonstration Research Corporation (MDRC), www.mdrc.org.
Kemple, J. J., & Willner, C. J. (2008a). Career Academies: Long-term impacts on labor market outcomes, educational attainment, and transitions to adulthood. Manpower Demonstration Research Corporation (MDRC), www.mdrc.org.
School: Low school commitment and attachment
Individual: Academic self-efficacy
Family: Parental involvement in education
School: Opportunities for prosocial involvement in education, Rewards for prosocial involvement in school
*
Risk/Protective Factor was significantly impacted by the program
See also: Career Academies Logic Model (PDF)
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
Study 1 (Kemple et al., 2000, 2004, 2008a, 2008b) found that the program was more effective for male than female participants, both in terms of labor market and social outcomes.
Sample, demographics including race, ethnicity, and gender for Blueprints-certified studies:
In Study 1, 44% of the subjects were male, 55% were Hispanic, 30% were Black, 6% were White, and 7% were Asian/Native American.
Training and technical assistance is offered in how to establish an individual Career Academy within a given high school or more broadly restructure one or more high schools around this approach. There is a wide range of services provided, and the cost depends on a number of factors, such as the size of the academy or high school or district, number of staff members to be involved in such training, where they are in the knowledge/implementation process (e.g., brand new, expanding), how much of what kinds of help is needed, travel costs, etc. The range of services provided is available on the website at: casn.berkeley.edu./services.php.
As a first step, it is recommended that interested persons start with a telephone conversation so that they can be directed to materials, all of which are free and available for downloading.
Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.
There are a number of national organizations that offer training and technical assistance to high schools interested in implementing a Career Academy; however, there is no required training or technical assistance in order to start an Academy. There are many guides and resources available on-line to help school staff interested in starting an Academy as well as many national organizations offering technical assistance. Some of these include: The College & Career Academy Support Network (CCASN), The National Academy Foundation (NAF), and the National Career Academy Coalition (NCAC). For information on other organizations offering technical assistance, see: http://casn.berkeley.edu/resources.php?r=242.
There is no single curriculum for Career Academies. The model requires academic courses connected to career and industry themes as well as career and technical education courses. There is a wide array of curricula available on-line. The College & Career Academy Support Network (CCASN) has an on-line database of curricula searchable by career/industry as well as academic subject area, http://casn.berkeley.edu/curriculum.php.
None.
Start-up of a Career Academy requires significant commitment and time from key administrators and teaching staff. The model requires: 1) a small learning community with a team of teachers working together to provide a supportive environment; 2) an integrated curriculum between academics and career-technical instruction; and 3) employer partnerships and involvement that allow students to have career experiences. Establishing a new Career Academy requires 8 months to 1 year of planning by an advisory board that includes industry partners from the career field, higher education representatives, teachers and administrators, community members, and parents.
No information is available
Ratios: The Career Academy model is implemented as a way of structuring learning in high schools so the teacher to student ratio is determined by the school district. However, a core element of the model is that students are organized into smaller learning communities within high schools, and participation by students is voluntary. A single Career Academy cluster within a high school could have 30-60 students per grade who take Academy classes together and are taught by a team of teachers. (Variations of the model in terms of the size of the learning community are being piloted in the interest of reaching a greater scale of students; however in the evaluations that have shown positive outcomes, the sizes of the learning clusters in the Academy model were in the 30-60 student range.)
Qualifications: The teaching and administrator qualifications required by the system would also apply to the Career Academy.
Time to Deliver Intervention: The core curriculum is delivered as part of the regular school day for high school students. The model does require designation of a lead teacher and a lead administrator who need to dedicate time to coordinating the Academy. The lead teacher should be provided at least a teaching period for planning and/or be reimbursed for time spent outside of the regular school day. Teachers' schedules should be structured so they have ongoing team planning and curricula time built in.
Costs that will vary by locality include administrative support; the staff time dedicated to outreach to employers, organizing and supporting students in internships and other career experiences; and space and equipment.
Ongoing training and technical assistance can be obtained from many national organizations working to support quality implementation of Career Academies and costs are negotiated individually with consultants.
There is no single required fidelity-monitoring tool. There are several sets of standards used in different settings. Perhaps the most widely applicable are those in the "Career Academy National Standards of Practice", available on both the CCASN (http://casn.berkeley.edu) and National Career Academy Coalition (NCAC) websites (www.ncacinc.com). NCAC has developed a set of rubrics for the standards of practice and will evaluate Academies for a fee. The National Academy Foundation (www.naf.org) also has such a set of standards for its 500 Academy National Network of Career Academies.
None.
No information is available
No information is available
In this example, one high school is implementing the Career Academy model with two cohorts including a total of 200 students. Costs are based on a study of costs from the California Multiple Pathways Program that includes Career Academy models (Ace Parsi, David Plank, and David Stern: Costs of California Multiple Pathway Programs. Policy Analysis for California Education (PACE), University of California, Berkeley, 2010). These are estimates; actual costs will vary depending on the local model and costs.
| Start-up ($297 per student) | $59,400.00 |
| Teacher and administrator planning time/coordination ($500 per student) | $100,000.00 |
| Total One Year Cost | $159,400.00 |
With 200 students participating, estimated year one costs per student would be $797 in addition to regular per pupil spending in the high school. Once the program was beyond start-up costs, the per-student cost would be closer to $500 in addition to per pupil spending.
The Career Academy is a school improvement approach that brings together a strong focus on both career and college readiness. Thus, it is generally implemented in place of other curricula and school improvement approaches and can be supported with the full range of federal, state, and local funds that support core K-12 education, as well as workforce development, and vocational and technical education funds.
High schools that implement a Career Academy will likely choose to shift funds spent on other curriculum and teacher professional development to this evidence-based model, as well as allocating teacher and administrator time to coordinate and implement the model.
State education funds allocated to local school systems as well as locally-appropriated public school funding can support Career Academies. Some states, for example California and Florida, have created state grant programs dedicated to supporting the Career Academy model.
Formula Funds:
Discretionary Grants: Federal discretionary grants from the U.S. Department of Education and the U.S. Department of Labor can be used to fund the initial training, ongoing staffing and coordination, technical assistance, and classroom materials. Relevant discretionary grants include the Career and Technical Education discretionary grants from the U.S. Department of Education, and the Department of Labor Youth Career Connect discretionary grants.
Because of the focus on a selected career or field, partnerships with employers are critical to the success of the model. Industry partners provide important in-kind resources including staff time for mentoring and guiding field trips of students; equipment and materials; staff time for organizing and supervising internships or job shadowing; and staff time for participation in the advisory group. Foundations, especially those with a stated interest in improving educational achievement and career outcomes for disadvantaged youth can also provide funding for training, coaching, technical assistance, staffing, and materials.
While support for the model is most typically taken on as a school system responsibility, fundraising can provide additional flexible dollars, especially when the school has many competing needs and priorities. Parent Teacher Associations, business and local civic associations can potentially serve as sponsors of fundraising campaigns, and can also provide important volunteer support.
College and Career Academy Support NetworkUniversity of California Berkeley1608 Tolman HallBerkeley, CA 94720-1670510-643-5748 casn.berkeley.edu
School-based programs designed to provide students with the credentials and skills needed to make successful transitions to post-secondary education and careers.
High-school students.
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
Study 1 (Kemple et al., 2000, 2004, 2008a, 2008b) found that the program was more effective for male than female participants, both in terms of labor market and social outcomes.
Sample, demographics including race, ethnicity, and gender for Blueprints-certified studies:
In Study 1, 44% of the subjects were male, 55% were Hispanic, 30% were Black, 6% were White, and 7% were Asian/Native American.
School: Low school commitment and attachment
Individual: Academic self-efficacy
Family: Parental involvement in education
School: Opportunities for prosocial involvement in education, Rewards for prosocial involvement in school
*Risk/Protective Factor was significantly impacted by the program
Career Academies are school-based programs that seek to reduce dropout rates and improve school performance and career readiness among high-school youth. A Career Academy (CA) is organized as a school-within-a-school, where students work in "small learning communities." Each small learning community involves a small number of students working with the same group of teachers for three or four years of high school with the aim being to create a more personalized and supportive learning environment for students. CAs offer students a combination of academic and career-technical curricula and use a career theme to integrate the two. In an effort to build connections between school and work and to provide students with a range of career development and work-based learning opportunities, CAs establish partnerships with local employers. To encourage post-graduate education they also build linkages to local colleges through curricular articulation, dual enrollment programs, and field trips to 2- and 4-year institutions.
A Career Academy is a personalized small learning community within a high school, comprised of a subset of students and teachers for a two-, three-, or four-year span. The goals of the Career Academy are to enhance students' engagement and performance in high school and provide them with the credentials and skills needed to make successful transitions to post-secondary education and, eventually, a career. Students enter an academy in 9th or 10th grade through a voluntary process; they must apply and be accepted, with parental knowledge and support. As academies function within the larger high school, they require a great deal of administrator and counselor support. Academies vary in size, usually including one to three sections of students at each grade level, or 100-300 students in all. Academy classes are generally blocked back-to-back in the daily schedule, and students attend them as a group. Students are able to complete academy requirements within the regular school day, with the exception of work internships and possible college classes.
An academy involves teachers from different subject areas working together as a team. This team manages the program, with one member usually serving as the coordinator. Teams usually participate in professional development, particularly in implementing the key features of the learning model and gaining exposure to the career field. Team members have shared planning time and usually a daily common planning period. The joining of a group of students for several periods each day with teachers whom they have come to know well provides a family-like atmosphere and serves to nurture close student-teacher ties. Academy students may also participate in required and elective classes outside the academy, as well as other activities such as clubs and sports.
Students in a Career Academy have a mixture of career classes (usually one or two) and academic classes (usually three or four) at a time. The academic and some of the career-technical classes meet entrance requirements for four-year colleges and universities. They are linked to academic and industry standards and encourage high achievement. They show students how their subjects relate to each other and the career field.
The career classes develop knowledge in a given field. They are designed to expose students to the full range of careers in that field. Special projects require students to bring together academic skills across their subjects and apply these to community and work settings outside the school. Usually the junior and/or senior year includes work experience, a paid or unpaid work internship or community service assignment. During the senior year students are provided with college and career counseling, forming a post-graduate plan which may include college, a mixture of work and college, or full-time work. The majority of graduates do attend college at some level, most commonly at two-year institutions.
The academy career theme is selected locally, based on an industry that is healthy and can provide a cadre of partners interested in supporting the program. Employers from a group of companies in the selected field work as partners in the academy, serving on a steering committee (along with teachers, administrators, and often parents and students) that governs the program's development and operation. This committee helps to plan the various activities in which employee volunteers participate: as speakers at the school, informing students of the industry and career options; as field trip and job shadowing hosts at their companies; as individual mentors, career-related "big brothers and sisters;" as work internship supervisors during the summer following the junior year or part-time during the senior year; and as community service coordinators. The employer partners may also hire graduates. Postsecondary educational institutions are usually included as well, providing course articulation and concurrent enrollment options.
Provides interpersonal support in the school environment, basically creating a school within a school.
Primary Evidence Base for Certification
Of the three studies Blueprints has reviewed, one (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness). The study was conducted by independent evaluators.
Study 1
Kemple et al. (2000, 2004, 2008a, 2008b) used a large scale, multi-site, random assignment research design. The researchers studied nine CAs with a sample size of 1,764 students. Students in the sample were identified in the 8th or 9th grade and were followed through their senior year of high school and up until eight years post-graduation. Data utilized in the study consisted of survey information provided by both Academy and non-Academy students in the study sample, performance indicators obtained from school records and transcripts, and standardized test scores from a test the researchers administered to the sample of students.
Primary Evidence Base for Certification
Study 1
Kemple et al. (2000, 2004, 2008a, 2008b) found that Career Academies reduced dropout rates and improved school engagement among students least likely to do well in a regular school environment (12th grade survey). However, over the longer term (4 and 8 years post-graduation), the program had no impact on high school completion rates, postsecondary enrollment, and postsecondary attainment. While the academies produced more modest effects for other students, they created a more supportive school environment for all students, and provided them with more opportunities to explore careers and engage in work-based learning opportunities.
Over the long term, students who attended Career Academies averaged 11% more in earnings over the eight years after graduation than those students in the non-Academy group. The Career Academies also produced an increase in the percentage of young people living independently with children and a spouse/partner.
Primary Evidence Base for Certification
Study 1
Kemple et al. (2000, 2004, 2008a, 2008b) found at the 12th grade survey that:
8 Years After Students' Scheduled Graduation:
One study meets Blueprints standards for high-quality methods with strong evidence of program impact (i.e., "certified" by Blueprints): Study 1 (Kemple et al., 2000, 2004, 2008a, 2008b). The sample for the study included students in grades eight and nine who had applied to one of the nine Career Academy sites used in the evaluation.
Study 1 took place in or near large urban school districts in California with above-average dropout rates, unemployment rates, and low-income families.
Additional Studies (not certified by Blueprints)
Study 2 (Bradby et al., 2007; Dayton et al., 2011)
Bradby, D., Dayton, C., Hanna, T., & Malloy, A. (2007). A profile of the California Partnership Academies, 2004-2005. Berkeley, CA: ConnectEd, and Career Academy Support Network, Graduate School of Education, University of California.
Dayton, C., Hester, C. H., & Stern, D. (2011). Profile of the California Partnership Academies 2009-2010. Career Academy Support Network, University of California Berkeley, and California Department of Education.
Study 3 (Hemelt et al., 2019)
Hemelt, S. W., Lenard, M. A., & Paeplow, C. G. (2019). Building bridges to life after high school: Contemporary career academies and student outcomes. Economics of Education Review, 68, 161-178. https://doi.org/10.1016/j.econedurev.2018.08.005
Blueprints: Promising
Crime Solutions: Effective
OJJDP Model Programs: Effective
Social Programs that Work:Top Tier
What Works Clearinghouse: Meets Standards Without Reservations - Positive Effect
Erin Fender
College and Career Academy Support Network
University of California
Berkeley Graduate School of Education
Berkeley, CA
Email: efender@berkeley.edu
(707) 591-6375
http://casn.berkeley.edu/index.php
Certified Kemple, J. J. (2004). Career Academies: Impacts on labor market outcomes and educational attainment. Manpower Demonstration Research Corporation (MDRC), www.mdrc.org.
Certified Kemple, J. J., & Snipes, J. C. (2000). Career Academies: Impacts on students' engagement and performance in high school. Manpower Demonstration Research Corporation (MDRC), www.mdrc.org.
Certified Kemple, J. J., & Willner, C. J. (2008a). Career Academies: Long-term impacts on labor market outcomes, educational attainment, and transitions to adulthood. Manpower Demonstration Research Corporation (MDRC), www.mdrc.org.
Kemple, J. J., & Willner, C. J. (2008b). Technical resources for Career Academies: Long-term impacts on labor market outcomes, educational attainment, and transitions to adulthood. Manpower Demonstration Research Corporation (MDRC), www.mdrc.org.
Bradby, D., Dayton, C., Hanna, T., & Malloy, A. (2007). A profile of the California Partnership Academies, 2004-2005. Berkeley, CA: ConnectEd, and Career Academy Support Network, Graduate School of Education, University of California.
Dayton, C., Hester, C. H., & Stern, D. (2011). Profile of the California Partnership Academies 2009-2010. Career Academy Support Network, University of California Berkeley, and California Department of Education.
Hemelt, S. W., Lenard, M. A., & Paeplow, C. G. (2019). Building bridges to life after high school: Contemporary career academies and student outcomes. Economics of Education Review, 68, 161-178. https://doi.org/10.1016/j.econedurev.2018.08.005
Summary
Kemple et al. (2000, 2004, 2008a, 2008b) used a large scale, multi-site, random assignment research design. The researchers studied nine CAs with a sample size of 1,764 students. Students in the sample were identified in the 8th or 9th grade and were followed through their senior year of high school and up until eight years post-graduation. Data utilized in the study consisted of survey information provided by both Academy and non-Academy students in the study sample, performance indicators obtained from school records and transcripts, and standardized test scores from a test the researchers administered to the sample of students.
Kemple et al. (2000, 2004, 2008a, 2008b) found at the 12th grade survey that:
8 Years After Students' Scheduled Graduation:
Evaluation Methodology
Design: A large scale, multi-site, random assignment research design was utilized to evaluate the effectiveness of Career Academies (CAs) in achieving their goals. Students in the sample (n=1764) were identified at the end of 8th or 9th grade and were followed through their senior year of high school, and up until eight years following their expected graduation. The treatment group (n=959) was comprised of students who applied to one of the 9 academies and who were randomly selected to participate. Students who applied to CAs but were randomly turned down served as the comparison group (n=805). Students were identified as either posing a high-risk (25% of sample), medium-risk (50%), or low-risk (25%) for dropping out of school, based on selected background characteristics and prior school experiences.
Data on outcomes were collected at three time points: posttest (at the time of expected graduation and the program end), 4 years after posttest, and 8 years after posttest. The posttest data came from a student survey, school records, and achievement tests while the follow-up data came from surveys. Long-term data from the two follow-up surveys cover the first four years and second four years post-graduation.
In order for schools to be eligible for the study, they had to have implemented and sustained the core features of the Career Academy approach - small learning communities, career-themed curricula, and partnerships with employers - for at least two years prior to commencement of the study.
The sample of 1,764 subjects from nine schools came from an initial sample of 1,953 students from 10 schools (Kemple & Snipes, 2000, p. 20, footnote 1). However, one of the original 10 schools dropped the program after 2 years and "was not able to provide follow-up information needed for the analyses in this report." The study did not appear to follow the 126 students in the school. In addition, 59 of the initial group of students were dropped "because they should not have been included in the study sample" and four students were found to be deceased.
Attrition. At posttest, response rates were 82% for school records data and 85% for the 12th-grade survey. The study also administered achievement tests to the 691 students in the study sample who were scheduled to be in 12th grade at the end of the 1997-98 school year (Kemple & Snipes, 2000, p. 18, footnote 24). The 490 students who completed the achievement tests represent 71% of those requested to take the tests but only 28% of the full sample.
At the 4-year follow-up, survey data was collected from 1,458 youth who completed the survey. This represents 83% of the 1,764 young people in the full study sample: 83% of the Academy group and 82% of the non-Academy group.
At the 8-year follow-up, survey data were collected from 1,428 youth. This represents 81% of the 1,764 young people in the full study sample: 82% of the Academy group and 80% of the non-Academy group.
Sample: Forty-four percent (44%) of the subjects were male. Fifty-five percent (55%) were Hispanic, 30% Black, 6% White, and 7% Asian/Native American. The remainder (n=34) were unaccounted for in the demographic breakdown. Twenty-six percent (26%) of the subjects were categorized as "high-risk", with 50% as "medium-risk" and 24% as "low-risk".
Each of the nine schools was located in or near large urban school districts with above average dropout rates, unemployment rates, and low-income families.
Measures: Posttest outcome measurements included high school enrollment rates, credits earned and course-taking patterns, math and reading achievement test scores, use of non-school hours and involvement in negative risk-taking behaviors, and steps taken towards further education and work and plans for the future. The researchers obtained data for evaluation from three sources: 1) school records and transcripts, which included information about attendance, academic credits, and course-taking patterns (for the 12% dropping out, up to the time of dropping out); 2) student surveys that asked a wide range of questions about school experiences, employment and work-related experiences, extracurricular activities, preparation for college and post-secondary jobs, and plans for the future; and 3) standardized math computation and reading comprehension tests, which were administered to 490 students from the sample.
The follow-up studies included a diverse set of measures relating to labor market outcomes (monthly earnings, hours worked), educational attainment (high school completion, postsecondary enrollment, postsecondary attainment), family formation (living independently with spouse and children), and social adjustment (health insurance, welfare, registered to vote, drug use).
To divide the sample in three groups based on the risk for dropping out (high, medium, and low), the study used six measures obtained at baseline: daily school attendance, GPA, credits toward graduation, over age for grade, sibling who dropped out of high school, and two or more transfers across schools. The six measures were used to predict the probability of dropping out, and the probability was used to define the three risk groups.
Additionally, to study the cross-site variation in the impacts of CAs, each CA was identified to be a low-, medium-, or high-contrast site. A high-contrast CA is a site where the school environment is most different from a non-CA site. Likewise, a low-contrast CA is a site where the school environment is not very different from a non-CA site. The primary construct used to distinguish among sites in the study was the difference in the level of interpersonal supports that CA and non-CA students received. A CA site was ranked according to the difference between the percentage of CA and non-CA students who reported a high level of support.
Analysis: The posttest used multiple regression analysis with controls for background characteristics. Results were presented separately for each of the three risk groups and for males and females. In the long-term analysis, estimates were regression adjusted to control for background characteristics, clustering of students within schools, and cluster within random assignment years. The studies reported .10 significance levels, but indicated the level of significance so that it is possible to identify condition differences at the .05 probability level or lower.
Many outcomes relating to the labor market, educational attainment, and family life could not be measured or controlled at baseline. However, the models used an extensive set of other baseline controls.
Concerning intent to treat, the study dropped 126 students who were part of the original sample of 1,953 but attended a school that disbanded the program. It is not clear if these students could have been followed and included in the study. Otherwise, the study included all students with data in both conditions, even those assigned to the intervention who never enrolled in or left the program (42%) and those assigned to the control group who enrolled in the intervention (6%). A statistic based on the impact per enrollee was also presented to show the effect on those who actually enrolled in the intervention.
Outcomes
Implementation fidelity: One of the eligibility criteria for study schools was that they had successfully maintained the core components of the Career Academy approach for at least two years before the study commenced. This suggests the programs were well developed. However, the difference in the results between high-contrast schools and low-contrast schools suggests that some schools deviated substantially from the program ideal.
Baseline equivalence: Table 2.1 (Kemple & Snipes, 2000, p. 25) shows no statistically significant differences between the background characteristics and prior school experiences of students in the academy and non-academy groups. The comparison included 26 measures of demographic and family characteristics, school performance characteristics, and characteristics associated with dropping out of school. There were also no significant differences on these background characteristics within the high-risk, medium-risk, and low-risk subgroups (Kemple & Snipes, 2000, Appendix B). A regression on the probability of being assigned to the intervention found only one significant variable for the full sample: intervention subjects tended to be older at baseline than control subjects.
Differential attrition: At posttest, response rates were similar across the intervention and control groups: 81.5% and 83.5% for the school records data, 86.2% and 84.8% for the 12th-grade survey, and 71.8% and 69.9% for those requested to take the achievement tests.
To examine differential attrition for background characteristics, regressions tested for the probability of being assigned to the intervention group among the analysis samples (i.e., the 81% with school records data, the 85% completing the survey data, and the 28% for the achievement test data). Only one of the background variables was associated significantly with being in the intervention group: the intervention group was older than the control group for the survey data analysis sample.
For the 4-year follow-up, the study found some differences between respondents and non-respondents (Kemple, 2004, Unit 1, p. 2): "Young men and high-risk students were somewhat underrepresented in the respondent sample, while young women and low-risk students were slightly overrepresented." However, the loss of subjects was similar across conditions. Comparison of baseline characteristics for the respondents in the analysis sample showed no significant differences between those in the intervention and control groups (Kemple, 2004, Exhibits 1.3 and 1.4).
For the 8-year follow-up, the analysis sample did not differ systematically on any of the background characteristics (Kemple & Willner, 2008a, p. 7). Details are found in the accompanying technical report (Kemple & Willner, 2008b). Exhibit 1.2 shows significantly lower response rates among males, older youth, students living in single parent households, mobile youth, youth missing classes, low attendance, poor academic performance, and risk of dropping out. However, the loss of subjects was similar across conditions. Exhibit 1.3 compares conditions for the analysis sample, finding a significant difference only in hours spent watching TV.
Posttest: The posttest examined condition differences for 50 outcomes within three groups - high-risk, medium-risk, and low-risk students - or 150 total tests. To summarize, the intervention group showed significantly better outcomes on 15 of the 50 tests for the high-risk students, 3 of 50 tests for the medium-risk students, and 5 of 50 tests for the low-risk students.
The treatment group experienced a statistically significant increase in their reported level of interpersonal support from faculty as well as an increase in their participation in career awareness and work-based learning activities. Among treatment group students with a high risk of dropping out, there was a significant reduction in dropout rates, improvement in attendance, increase in academic course-taking, and increase in the likelihood of earning enough credits to graduate on time. Among treatment group students with a low risk of dropping out, there was an increase in the likelihood of graduating on time. Treatment group students across risk levels experienced an increase in vocational course-taking without reducing their likelihood of completing a basic core academic curriculum. When findings were averaged across groups, there were only slight reductions in dropout rates and modest increases in other measures of school engagement, but the only significant impact was for more course credits.
High-contrast CAs, those intervention sites where the school environment is very different from non-CA schools, produced a statistically significant lower dropout rate among medium-risk students, as well as an increase in completion of a core academic curriculum. Conversely, low-contrast CAs produced higher dropout rates, reduced attendance, and lower rates of academic course-taking for medium-risk students.
Long-term (48 months post-graduation): In terms of the full sample, CA students earned significantly more ($107, p<0.05) per month than their control group counterparts at the 48-month follow-up. CA students also averaged 1.9 more working hours per week (p<0.05) and $0.47 more per hour than non-academy students (p<0.05). The program had no effect on high school completion rates, postsecondary enrollment, and postsecondary attainment, but the authors note that the program focus on careers did not worsen academic success. A summary of the results for the full sample in Exhibits 3.1, 3.6, and 3.9 of Kemple (2004) shows that 5 of 43 tests reached statistical significance at the .05 level.
When analyzed by subgroup, it was young men, rather than young women who experienced the greatest change due to the program. The male subgroup average monthly earnings were $212 more than their control group counterparts. This was reflected in significantly greater months of employment over the four years (2.8), greater hours worked per week (4.2), and higher hourly wages ($0.74). In terms of the low-, medium-, and high-risk subgroups, the medium risk group was the only academy group to have significantly higher monthly ($141) and hourly ($0.53) earnings than their non-academy equivalents.
One possible iatrogenic effect identified was that male and high-risk students within the Career Academy group were less likely than their non-Academy counterparts to have enrolled in post-secondary education. However, the lower enrollment did not adversely affect labor market outcomes.
Note: The report at 8-years post-graduation indicates slightly varied results in terms of the outcomes for years 1-4 compared to the original report produced at year 4. See Kemple & Willner (2008a, p. 13) for results over the full period.
Long-term (96 months post-graduation): For the full sample over years 5-8 post-graduation, intervention subjects earned $216 more (p<0.01) per month than their control group counterparts. They also worked on average 1.7 hours more per week (p<0.05). With regard to social outcomes, the academy group had higher levels of being a custodial parent (p=0.006), lower levels of being a noncustodial parent (p=0.026) and higher levels of living independently with children and a partner (p=0.01). The program had no effect on high school completion rates, postsecondary enrollment, or postsecondary attainment, but the authors note that the program focus on careers did not worsen academic success. A summary of the results for the full sample in Exhibits 3.1B, 3.4, and 3.6 in Kemple & Willner (2008b) shows that 7 of 40 tests reached statistical significance at the .05 level.
When analyzed by subgroup, young men again experienced the greatest labor-related outcomes from the program. Young men from the academy group earned on average $361 (p<0.05) more per month than their control group counterparts. They also were employed for 2.8 months more over the 4-year period (p<0.01) and worked 4.1 hours more per week (p<0.01). With regards to social outcomes, male academy group participants were more likely to be married and living with their partner (p=0.020), more likely to be a custodial parent (p=0.003), less likely to be a noncustodial parent (p=0.023), and less likely to be living independently with no children (p=0.015). Female academy group participants were less likely to be living with parents/guardians with or without children (p=0.005).
In terms of the high-, medium-, and low-risk subgroups, the analyses of monthly earnings showed no significant differences across the three subgroups in the effect of the program. More detailed analyses for the subgroups are presented in the technical report (Kemple & Willner 2008b). There were also no differences in the null effect of the program on educational outcomes across the three subgroups.
Summary
Bradby et al. (2007) and Dayton et al. (2011) compared students in 467 California Partnership Academies, which used the Career Academy model, to students statewide. The design involved no assignment, matching, or comparisons of students over time.
Bradby et al. (2007) and Dayton et al. (2011) found that, compared to California students statewide, students attending California Partnership Academies showed higher
Evaluation Methodology
California Partnership Academies use the Career Academies model, usually in grades 10-12. Bradby et al. (2007) present data for 2004-2005, but Dayton et al. (2011) incorporate the earlier study in examining data for 2004-2005 and 2009-2010.
Design:
This study presents a profile of schools and students in California Partnerships Academies (CPAs), with some comparison to statistics for the state of California as a whole. It did not assign or match schools or students to treatment and control groups. Given that students self-selected into the program schools and likely differed substantially from other students in motivation, skills, and background characteristics, the study cannot establish the causal impact of the program.
The study obtained data in 2004-2005 and 2009-2010 from the California Department of Education on schools and students from 467 CPAs that operated in 278 of California's 1,264 comprehensive high schools. The schools are located in 36 of California's 58 counties, with the largest concentration in the six most populous counties. The 467 CPAs enrolled 48,436 students in grades 10-12 - about 3% of all California students in those grades.
Each CPA submits an annual report to the California Department of Education. The study used data from these annual reports on students in grades 10-12. The cross-sectional comparisons do not follow students or schools over time or compare outcomes for higher grades to lower grades.
Sample: By law, at least 50% of the students in each incoming class of CPA sophomores must meet three of six at risk-requirements (e.g., poor attendance record, low motivation for school, low test scores). Most (54%) of the schools fell in the bottom 40% of schools statewide. The students were 53% female and 47% male and were primarily Hispanic (59%), but also white (16%), Asian (10%), and black (9%).
Measures: CPAs must submit annual reports on how many students meet specified targets for attendance, credits, and graduation. Under the performance-based program, CPAs receive funding only for students who meet the targets. The measures depend on the accuracy of reports from the schools, which deliver the program and have financial incentives to show positive results.
Analysis: The report listed percentages for CPAs and state schools without tests of significance, controls for baseline outcomes, or adjustments for clustering within schools. Along with overall percentages, the report presented figures for CPA and state schools within gender and race/ethnicity groups.
Outcomes
Implementation Fidelity: Fidelity showed in high percentage of CPA seniors having postsecondary plans, working with a mentor, and participating in work-based learning. Older academies did better on test scores than new academies, likely because they had more time to implement the program well.
Baseline Equivalence: The distribution by race and ethnicity in the CPAs differed from that in the state overall. Hispanics made up 59% of CPAs and 47% of schools statewide; whites made up 16% of CPAs, and 30% of schools statewide. CPAs had slightly more females than schools statewide, 53% versus 49%. The study presented no other tests for preexisting student differences.
Differential Attrition: Since subjects were not followed over time, the study could not test for differential attrition.
Posttest: Comparison of outcomes with data for both CPAs and state schools showed the following results.
Pass Rates for Math and English Language Arts. Test results for the 10th-grade California High School Exit Exam demonstrated higher pass rates for CPAs than schools statewide in both math (80% versus 74%) and English (84% versus 76%) in 2004-2005 but little difference in 2009-2010 (83% versus 81% in math and 82% versus 81% in English).
Graduation Rates. The outcome measures the percentage receiving a diploma among those who remained in high school until their senior year (but doesn't account for potentially different dropout rates before the senior year). On this measure, CPAs did better than statewide schools in 2004-2005 (96% versus 87%) and 2009-2010 (95% versus 85%).
Meeting California College Course Requirements. Higher percentages of CPA students than students statewide completed course requirements for entrance into the University of California and California State colleges in 2004-05 (50% versus 35%) and 2009-2010 (57% versus 36%).
Modifiers. The differences between CPAs and all state schools appear similar for males and females, but males perhaps benefitted more from the program. The differences among race/ethnic groups also appear similar, but students of color may have benefitted more from the program. There is some evidence that Asians did worse in CPA schools. Also, CPA schools with a focus on manufacturing and product development had higher exam pass rates (100%) than CPA schools with a focus on marketing sales and service (67%) and building trades and construction (57%).
Summary
Hemelt et al. (2019) used a quasi-experimental, instrumental variable design based on both the actual participation in the program and an instrument measuring randomized assignment into the program. A total of 469 9th grade students in a single North Carolina high school comprised the analysis sample. Random assignment to groups in a career academy assignment lottery was used to predict treatment enrollment, which was in turn used to estimate the causal effect of enrolling in the treatment. High school graduation rates, reading and math test scores, college enrollment, attendance, industry certification, and course-taking were assessed at baseline and follow-up.
Hemelt et al. (2019) found that, relative to the control group, enrollment in the career academy intervention group significantly
Evaluation Methodology
Design:
Recruitment: The sample included first-time 9th grade students in Wake County, North Carolina, who entered high school in 2009-2010 to 2012-2013. Students in the initial sample (n=646) attended a single school that had both a traditional high school curriculum and career academy curricula focused on Information Technology (IT) and Science, Technology, Engineering, and Math (STEM). Applicants were excluded if they had 1) siblings already enrolled in the career academy (n=136 students) because they were guaranteed admission, or 2) missing baseline test score or demographic information (n=41 students). The exclusions resulted in a final analytic sample of 469 students.
Assignment: The study used a quasi-experimental, instrumental variable design based on both the actual participation in the program and an instrument measuring randomized assignment into the program.
First, the initial randomization to the conditions defined an instrumental variable that was used to adjust the actual program participation for selection bias. Group assignment was based on a lottery (applications submitted at the end of 8th grade) for admission into the career academy housed within one high school. The authors noted that the majority of students who lost the lottery (92%) enrolled in the regular track (i.e., non-career academy program) at the same high school.
Second, actual participation or non-participation in the program defined the two conditions. Since students who lost the lottery could still enroll in the academy from the waitlist if open seats remained after being offered to lottery winners, the randomization did not completely define condition participation. Both intervention and control group students attended the same school (Apex High School). The intervention group included students who won the lottery and enrolled in the Academy of Information Technology (AOIT), which included a technology-based paid internship, a four-year sequence of IT courses/electives, cohort-based grouping with fellow program enrollees, career development, soft-skills training throughout the four-year program, completion of one of two curricular IT tracks and district-wide elective requirements. As with admission into the academy itself, both track and suggested elective courses based on track were first filled with AOIT students and, if seats remained open, non-AOIT students were then eligible to enroll. The control group included students who lost the lottery and enrolled in the "traditional" high school curriculum track. Neither the number of lottery winners vs. losers nor the number of enrollee compliers vs. non-enrollee compliers were provided.
Assessments/Attrition: The study used administrative data, but attrition was not reported. In the Table A2 table note, the authors stated, "In 2014-15, a second career academy to which students zoned for Apex High School could apply opened in Apex Friendship High School (Academy of Engineering and Advanced Manufacturing) and thus the share of students who lost the AOIT lottery and attended 'regular' Apex High School dropped because many entered the new academy." This statement suggests that there may have been some attrition over the course of the study period.
Sample:
The analytic sample of lottery applicants (n=469) included students who were 36% female, 78.7% non-Hispanic White, 10.4% Asian, 5.1% non-Hispanic Black, and 2.6% Hispanic. Additionally, 8.3% were involved in special education, 57.9% had an academically gifted status, and had 8th grade standardized academic achievement scores of 0.745 for math and 0.738 for reading.
Measures:
Study measures came from official administrative data housed by the North Carolina Department of Public Instruction, which includes student-level demographic data (i.e., gender, race/ethnicity, special education status, and limited English proficiency [LEP] status), academic engagement and achievement data (e.g., standardized reading and math test scores), and college enrollment data using the National Student Clearinghouse.
The primary outcome was on-time high school graduation (i.e., any time by the end of the fourth year of high school). Secondary behavioral outcomes were attendance, academic performance in high school, and college enrollment. Additional intermediate outcomes were AP course-taking and industry certifications in high school, which represented risk and protective factors.
Analysis:
The researchers conducted two sets of analyses. The first set estimated the effects of winning the lottery on several high school and college outcomes (i.e., intent-to-treat analyses). The second, primary set of analyses used an instrumental variables approach to estimate the average effect of enrolling in the career academy program among students who did so because they were lottery winners on the same outcomes, which is the local average treatment effect (LATE). The instrument was a student's "random" assignment from the lottery, which was used to predict enrollment into the career academy, which was in turn used to estimate the causal effect of the treatment on the outcome variables. The models used two-stage least squares estimation and included pre-lottery covariates (e.g., gender, race/ethnicity, prior academic achievement) and lottery-cohort fixed effects. The authors also conducted a treatment-on-the-treated (TOT) analysis for the attendance outcome.
Exploratory subgroup analyses examined whether program impacts differed by gender and baseline achievement levels (i.e., 8th grade test scores). Additional intermediate outcome analyses examined whether attendance in 9th grade, academic performance and AP course-taking during high school, and industry-recognized credential attainment mediated high school completion.
Intent-to-Treat: The first set of analyses that estimated the effects of winning the lottery on outcomes of interest followed an intent-to-treat approach. Although the instrumental variable approach (i.e., the second, primary set of analyses) also included all participants with complete data, it may violate the intent-to-treat criterion by examining the effects of actual participation in the program rather than the effects of random assignment to the program. However, unlike a treatment-of-the treated analysis, the instrumental variable approach minimizes rather than increases selection bias.
Missing Data Method: The analysis included participants with complete data at baseline and follow-up. The authors also conducted sensitivity analyses, described in a footnote on page 168, stating, "Results are nearly identical and conclusions unchanged if we keep these observations in the analytic dataset and control for missing information using indicator variables in our regressions."
Outcomes
Implementation Fidelity: Not reported.
Baseline Equivalence:
The study did not compare the conditions directly but conducted tests for baseline equivalence on the randomized lottery groups. There were no statistically significant differences between students who won the lottery and those who lost the lottery on demographic characteristics or baseline academic achievement outcomes (Table 4).
Differential Attrition:
The study used administrative data, but there were no details about missing student graduation data or other outcome data. However, the authors conducted tests comparing the baseline characteristics of students who won vs. lost the lottery and reported no significant differences (see p. 168 text and Table 4).
Posttest:
Career academy enrollment significantly increased the likelihood of expected, on-time high school graduation by about 8 percentage points. TOT analyses revealed that career academy participation also significantly reduced 9th grade absences by about 1.4 days. Both of these effects remained significant in ITT analyses that examined differences in program impacts between students who won the lottery and those who lost the lottery. However, academy participation did not have a significant effect on academic performance (reading, math, and composite test scores) or college enrollment.
For the intermediate outcomes, career academy participation significantly increased the likelihood of earning an IT industry certification. Academy participation did not have a significant effect on rates of AP course-taking during high school.
Subgroup analyses showed that the overall effect on high school graduation was driven by impacts on males as well as students in the middle to upper-middle of the baseline math achievement distribution. Participation in the academy increased the likelihood of college enrollment within one year of expected, on-time high school graduation for males but not females.
Long-Term: Not tested, as the program extended through high school.