Career Academies

Study 1

Summary

Kemple et al. (2000, 2004, 2008a, 2008b) used a large scale, multi-site, random assignment research design. The researchers studied nine CAs with a sample size of 1,764 students. Students in the sample were identified in the 8th or 9th grade and were followed through their senior year of high school and up until eight years post-graduation. Data utilized in the study consisted of survey information provided by both Academy and non-Academy students in the study sample, performance indicators obtained from school records and transcripts, and standardized test scores from a test the researchers administered to the sample of students.

Kemple et al. (2000, 2004, 2008a, 2008b) found at the 12th grade survey that:

• Career Academies (CAs) increased both the level of interpersonal support students experienced during high school and their participation in career awareness and work-based learning activities.
• Among students with a high-risk of dropping out, CAs reduced dropout rates, improved attendance, increased academic course-taking, and increased the likelihood of earning enough credits to graduate on time.
• Among students with a low-risk of dropping-out, CAs increased the likelihood of graduating on time. CAs also increased career-related course-taking for these students without reducing their likelihood of completing a basic core academic curriculum.
• In sites where CAs produced particularly dramatic enhancements in the interpersonal support that students received from teachers and peers, the CAs reduced dropout rates and improved school engagement for both high-risk and medium-risk subgroups. CAs that did not enhance these supports actually increased dropout rates and reduced school engagement for some students.
• CAs did not improve standardized math and reading achievement scores.

8 Years After Students' Scheduled Graduation:

• CAs students averaged 11% more in earnings than their non-CA group counterparts; when disaggregated by sex, significant effects were found for males only.
• CAs students worked 1.7 hours more per week.
• CAs students had a significantly higher percentage of living independently with children and a spouse/partner than non-Career Academy students, in the eight years following graduation.
• No effects on high school completion rates, postsecondary enrollment, and postsecondary attainment.

Evaluation Methodology

Design: A large scale, multi-site, random assignment research design was utilized to evaluate the effectiveness of Career Academies (CAs) in achieving their goals. Students in the sample (n=1764) were identified at the end of 8th or 9th grade and were followed through their senior year of high school, and up until eight years following their expected graduation. The treatment group (n=959) was comprised of students who applied to one of the 9 academies and who were randomly selected to participate. Students who applied to CAs but were randomly turned down served as the comparison group (n=805). Students were identified as either posing a high-risk (25% of sample), medium-risk (50%), or low-risk (25%) for dropping out of school, based on selected background characteristics and prior school experiences.

Data on outcomes were collected at three time points: posttest (at the time of expected graduation and the program end), 4 years after posttest, and 8 years after posttest. The posttest data came from a student survey, school records, and achievement tests while the follow-up data came from surveys. Long-term data from the two follow-up surveys cover the first four years and second four years post-graduation.

In order for schools to be eligible for the study, they had to have implemented and sustained the core features of the Career Academy approach - small learning communities, career-themed curricula, and partnerships with employers - for at least two years prior to commencement of the study.

The sample of 1,764 subjects from nine schools came from an initial sample of 1,953 students from 10 schools (Kemple & Snipes, 2000, p. 20, footnote 1). However, one of the original 10 schools dropped the program after 2 years and "was not able to provide follow-up information needed for the analyses in this report." The study did not appear to follow the 126 students in the school. In addition, 59 of the initial group of students were dropped "because they should not have been included in the study sample" and four students were found to be deceased.

Attrition. At posttest, response rates were 82% for school records data and 85% for the 12th-grade survey. The study also administered achievement tests to the 691 students in the study sample who were scheduled to be in 12th grade at the end of the 1997-98 school year (Kemple & Snipes, 2000, p. 18, footnote 24). The 490 students who completed the achievement tests represent 71% of those requested to take the tests but only 28% of the full sample.

At the 4-year follow-up, survey data was collected from 1,458 youth who completed the survey. This represents 83% of the 1,764 young people in the full study sample: 83% of the Academy group and 82% of the non-Academy group.

At the 8-year follow-up, survey data were collected from 1,428 youth. This represents 81% of the 1,764 young people in the full study sample: 82% of the Academy group and 80% of the non-Academy group.

Sample: Forty-four percent (44%) of the subjects were male. Fifty-five percent (55%) were Hispanic, 30% Black, 6% White, and 7% Asian/Native American. The remainder (n=34) were unaccounted for in the demographic breakdown. Twenty-six percent (26%) of the subjects were categorized as "high-risk", with 50% as "medium-risk" and 24% as "low-risk".

Each of the nine schools was located in or near large urban school districts with above average dropout rates, unemployment rates, and low-income families.

Measures: Posttest outcome measurements included high school enrollment rates, credits earned and course-taking patterns, math and reading achievement test scores, use of non-school hours and involvement in negative risk-taking behaviors, and steps taken towards further education and work and plans for the future. The researchers obtained data for evaluation from three sources: 1) school records and transcripts, which included information about attendance, academic credits, and course-taking patterns (for the 12% dropping out, up to the time of dropping out); 2) student surveys that asked a wide range of questions about school experiences, employment and work-related experiences, extracurricular activities, preparation for college and post-secondary jobs, and plans for the future; and 3) standardized math computation and reading comprehension tests, which were administered to 490 students from the sample.

The follow-up studies included a diverse set of measures relating to labor market outcomes (monthly earnings, hours worked), educational attainment (high school completion, postsecondary enrollment, postsecondary attainment), family formation (living independently with spouse and children), and social adjustment (health insurance, welfare, registered to vote, drug use).

To divide the sample in three groups based on the risk for dropping out (high, medium, and low), the study used six measures obtained at baseline: daily school attendance, GPA, credits toward graduation, over age for grade, sibling who dropped out of high school, and two or more transfers across schools. The six measures were used to predict the probability of dropping out, and the probability was used to define the three risk groups.

Additionally, to study the cross-site variation in the impacts of CAs, each CA was identified to be a low-, medium-, or high-contrast site. A high-contrast CA is a site where the school environment is most different from a non-CA site. Likewise, a low-contrast CA is a site where the school environment is not very different from a non-CA site. The primary construct used to distinguish among sites in the study was the difference in the level of interpersonal supports that CA and non-CA students received. A CA site was ranked according to the difference between the percentage of CA and non-CA students who reported a high level of support.

Analysis: The posttest used multiple regression analysis with controls for background characteristics. Results were presented separately for each of the three risk groups and for males and females. In the long-term analysis, estimates were regression adjusted to control for background characteristics, clustering of students within schools, and cluster within random assignment years. The studies reported .10 significance levels, but indicated the level of significance so that it is possible to identify condition differences at the .05 probability level or lower.

Many outcomes relating to the labor market, educational attainment, and family life could not be measured or controlled at baseline. However, the models used an extensive set of other baseline controls.

Concerning intent to treat, the study dropped 126 students who were part of the original sample of 1,953 but attended a school that disbanded the program. It is not clear if these students could have been followed and included in the study. Otherwise, the study included all students with data in both conditions, even those assigned to the intervention who never enrolled in or left the program (42%) and those assigned to the control group who enrolled in the intervention (6%). A statistic based on the impact per enrollee was also presented to show the effect on those who actually enrolled in the intervention.

Outcomes

Implementation fidelity: One of the eligibility criteria for study schools was that they had successfully maintained the core components of the Career Academy approach for at least two years before the study commenced. This suggests the programs were well developed. However, the difference in the results between high-contrast schools and low-contrast schools suggests that some schools deviated substantially from the program ideal.

Baseline equivalence: Table 2.1 (Kemple & Snipes, 2000, p. 25) shows no statistically significant differences between the background characteristics and prior school experiences of students in the academy and non-academy groups. The comparison included 26 measures of demographic and family characteristics, school performance characteristics, and characteristics associated with dropping out of school. There were also no significant differences on these background characteristics within the high-risk, medium-risk, and low-risk subgroups (Kemple & Snipes, 2000, Appendix B). A regression on the probability of being assigned to the intervention found only one significant variable for the full sample: intervention subjects tended to be older at baseline than control subjects.

Differential attrition: At posttest, response rates were similar across the intervention and control groups: 81.5% and 83.5% for the school records data, 86.2% and 84.8% for the 12th-grade survey, and 71.8% and 69.9% for those requested to take the achievement tests.

To examine differential attrition for background characteristics, regressions tested for the probability of being assigned to the intervention group among the analysis samples (i.e., the 81% with school records data, the 85% completing the survey data, and the 28% for the achievement test data). Only one of the background variables was associated significantly with being in the intervention group: the intervention group was older than the control group for the survey data analysis sample.

For the 4-year follow-up, the study found some differences between respondents and non-respondents (Kemple, 2004, Unit 1, p. 2): "Young men and high-risk students were somewhat underrepresented in the respondent sample, while young women and low-risk students were slightly overrepresented." However, the loss of subjects was similar across conditions. Comparison of baseline characteristics for the respondents in the analysis sample showed no significant differences between those in the intervention and control groups (Kemple, 2004, Exhibits 1.3 and 1.4).

For the 8-year follow-up, the analysis sample did not differ systematically on any of the background characteristics (Kemple & Willner, 2008a, p. 7). Details are found in the accompanying technical report (Kemple & Willner, 2008b). Exhibit 1.2 shows significantly lower response rates among males, older youth, students living in single parent households, mobile youth, youth missing classes, low attendance, poor academic performance, and risk of dropping out. However, the loss of subjects was similar across conditions. Exhibit 1.3 compares conditions for the analysis sample, finding a significant difference only in hours spent watching TV.

Posttest: The posttest examined condition differences for 50 outcomes within three groups - high-risk, medium-risk, and low-risk students - or 150 total tests. To summarize, the intervention group showed significantly better outcomes on 15 of the 50 tests for the high-risk students, 3 of 50 tests for the medium-risk students, and 5 of 50 tests for the low-risk students.

The treatment group experienced a statistically significant increase in their reported level of interpersonal support from faculty as well as an increase in their participation in career awareness and work-based learning activities. Among treatment group students with a high risk of dropping out, there was a significant reduction in dropout rates, improvement in attendance, increase in academic course-taking, and increase in the likelihood of earning enough credits to graduate on time. Among treatment group students with a low risk of dropping out, there was an increase in the likelihood of graduating on time. Treatment group students across risk levels experienced an increase in vocational course-taking without reducing their likelihood of completing a basic core academic curriculum. When findings were averaged across groups, there were only slight reductions in dropout rates and modest increases in other measures of school engagement, but the only significant impact was for more course credits.

High-contrast CAs, those intervention sites where the school environment is very different from non-CA schools, produced a statistically significant lower dropout rate among medium-risk students, as well as an increase in completion of a core academic curriculum. Conversely, low-contrast CAs produced higher dropout rates, reduced attendance, and lower rates of academic course-taking for medium-risk students.

Long-term (48 months post-graduation): In terms of the full sample, CA students earned significantly more ($107, p<0.05) per month than their control group counterparts at the 48-month follow-up. CA students also averaged 1.9 more working hours per week (p<0.05) and $0.47 more per hour than non-academy students (p<0.05). The program had no effect on high school completion rates, postsecondary enrollment, and postsecondary attainment, but the authors note that the program focus on careers did not worsen academic success. A summary of the results for the full sample in Exhibits 3.1, 3.6, and 3.9 of Kemple (2004) shows that 5 of 43 tests reached statistical significance at the .05 level.

When analyzed by subgroup, it was young men, rather than young women who experienced the greatest change due to the program. The male subgroup average monthly earnings were $212 more than their control group counterparts. This was reflected in significantly greater months of employment over the four years (2.8), greater hours worked per week (4.2), and higher hourly wages ($0.74). In terms of the low-, medium-, and high-risk subgroups, the medium risk group was the only academy group to have significantly higher monthly ($141) and hourly ($0.53) earnings than their non-academy equivalents.

One possible iatrogenic effect identified was that male and high-risk students within the Career Academy group were less likely than their non-Academy counterparts to have enrolled in post-secondary education. However, the lower enrollment did not adversely affect labor market outcomes.

Note: The report at 8-years post-graduation indicates slightly varied results in terms of the outcomes for years 1-4 compared to the original report produced at year 4. See Kemple & Willner (2008a, p. 13) for results over the full period.

Long-term (96 months post-graduation): For the full sample over years 5-8 post-graduation, intervention subjects earned $216 more (p<0.01) per month than their control group counterparts. They also worked on average 1.7 hours more per week (p<0.05). With regard to social outcomes, the academy group had higher levels of being a custodial parent (p=0.006), lower levels of being a noncustodial parent (p=0.026) and higher levels of living independently with children and a partner (p=0.01). The program had no effect on high school completion rates, postsecondary enrollment, or postsecondary attainment, but the authors note that the program focus on careers did not worsen academic success. A summary of the results for the full sample in Exhibits 3.1B, 3.4, and 3.6 in Kemple & Willner (2008b) shows that 7 of 40 tests reached statistical significance at the .05 level.

When analyzed by subgroup, young men again experienced the greatest labor-related outcomes from the program. Young men from the academy group earned on average $361 (p<0.05) more per month than their control group counterparts. They also were employed for 2.8 months more over the 4-year period (p<0.01) and worked 4.1 hours more per week (p<0.01). With regards to social outcomes, male academy group participants were more likely to be married and living with their partner (p=0.020), more likely to be a custodial parent (p=0.003), less likely to be a noncustodial parent (p=0.023), and less likely to be living independently with no children (p=0.015). Female academy group participants were less likely to be living with parents/guardians with or without children (p=0.005).

In terms of the high-, medium-, and low-risk subgroups, the analyses of monthly earnings showed no significant differences across the three subgroups in the effect of the program. More detailed analyses for the subgroups are presented in the technical report (Kemple & Willner 2008b). There were also no differences in the null effect of the program on educational outcomes across the three subgroups.

Study 2

Summary

Bradby et al. (2007) and Dayton et al. (2011) compared students in 467 California Partnership Academies, which used the Career Academy model, to students statewide. The design involved no assignment, matching, or comparisons of students over time.

Bradby et al. (2007) and Dayton et al. (2011) found that, compared to California students statewide, students attending California Partnership Academies showed higher

• Graduation rates
• Completion rates of course requirements for college entrance.

Evaluation Methodology

California Partnership Academies use the Career Academies model, usually in grades 10-12. Bradby et al. (2007) present data for 2004-2005, but Dayton et al. (2011) incorporate the earlier study in examining data for 2004-2005 and 2009-2010.

Design:

This study presents a profile of schools and students in California Partnerships Academies (CPAs), with some comparison to statistics for the state of California as a whole. It did not assign or match schools or students to treatment and control groups. Given that students self-selected into the program schools and likely differed substantially from other students in motivation, skills, and background characteristics, the study cannot establish the causal impact of the program.

The study obtained data in 2004-2005 and 2009-2010 from the California Department of Education on schools and students from 467 CPAs that operated in 278 of California's 1,264 comprehensive high schools. The schools are located in 36 of California's 58 counties, with the largest concentration in the six most populous counties. The 467 CPAs enrolled 48,436 students in grades 10-12 - about 3% of all California students in those grades.

Each CPA submits an annual report to the California Department of Education. The study used data from these annual reports on students in grades 10-12. The cross-sectional comparisons do not follow students or schools over time or compare outcomes for higher grades to lower grades.

Sample: By law, at least 50% of the students in each incoming class of CPA sophomores must meet three of six at risk-requirements (e.g., poor attendance record, low motivation for school, low test scores). Most (54%) of the schools fell in the bottom 40% of schools statewide. The students were 53% female and 47% male and were primarily Hispanic (59%), but also white (16%), Asian (10%), and black (9%).

Measures: CPAs must submit annual reports on how many students meet specified targets for attendance, credits, and graduation. Under the performance-based program, CPAs receive funding only for students who meet the targets. The measures depend on the accuracy of reports from the schools, which deliver the program and have financial incentives to show positive results.

Analysis: The report listed percentages for CPAs and state schools without tests of significance, controls for baseline outcomes, or adjustments for clustering within schools. Along with overall percentages, the report presented figures for CPA and state schools within gender and race/ethnicity groups.

Outcomes

Implementation Fidelity: Fidelity showed in high percentage of CPA seniors having postsecondary plans, working with a mentor, and participating in work-based learning. Older academies did better on test scores than new academies, likely because they had more time to implement the program well.

Baseline Equivalence: The distribution by race and ethnicity in the CPAs differed from that in the state overall. Hispanics made up 59% of CPAs and 47% of schools statewide; whites made up 16% of CPAs, and 30% of schools statewide. CPAs had slightly more females than schools statewide, 53% versus 49%. The study presented no other tests for preexisting student differences.

Differential Attrition: Since subjects were not followed over time, the study could not test for differential attrition.

Posttest: Comparison of outcomes with data for both CPAs and state schools showed the following results.

Pass Rates for Math and English Language Arts. Test results for the 10th-grade California High School Exit Exam demonstrated higher pass rates for CPAs than schools statewide in both math (80% versus 74%) and English (84% versus 76%) in 2004-2005 but little difference in 2009-2010 (83% versus 81% in math and 82% versus 81% in English).

Graduation Rates. The outcome measures the percentage receiving a diploma among those who remained in high school until their senior year (but doesn't account for potentially different dropout rates before the senior year). On this measure, CPAs did better than statewide schools in 2004-2005 (96% versus 87%) and 2009-2010 (95% versus 85%).

Meeting California College Course Requirements. Higher percentages of CPA students than students statewide completed course requirements for entrance into the University of California and California State colleges in 2004-05 (50% versus 35%) and 2009-2010 (57% versus 36%).

Modifiers. The differences between CPAs and all state schools appear similar for males and females, but males perhaps benefitted more from the program. The differences among race/ethnic groups also appear similar, but students of color may have benefitted more from the program. There is some evidence that Asians did worse in CPA schools. Also, CPA schools with a focus on manufacturing and product development had higher exam pass rates (100%) than CPA schools with a focus on marketing sales and service (67%) and building trades and construction (57%).

Study 3

Summary

Hemelt et al. (2019) used a quasi-experimental, instrumental variable design based on both the actual participation in the program and an instrument measuring randomized assignment into the program. A total of 469 9^th grade students in a single North Carolina high school comprised the analysis sample. Random assignment to groups in a career academy assignment lottery was used to predict treatment enrollment, which was in turn used to estimate the causal effect of enrolling in the treatment. High school graduation rates, reading and math test scores, college enrollment, attendance, industry certification, and course-taking were assessed at baseline and follow-up.

Hemelt et al. (2019) found that, relative to the control group, enrollment in the career academy intervention group significantly

• Increased high school graduation rates
• Increased the likelihood of earning an IT industry certification
• Reduced ninth-grade absences.

Evaluation Methodology

Design:

Recruitment: The sample included first-time 9th grade students in Wake County, North Carolina, who entered high school in 2009-2010 to 2012-2013. Students in the initial sample (n=646) attended a single school that had both a traditional high school curriculum and career academy curricula focused on Information Technology (IT) and Science, Technology, Engineering, and Math (STEM). Applicants were excluded if they had 1) siblings already enrolled in the career academy (n=136 students) because they were guaranteed admission, or 2) missing baseline test score or demographic information (n=41 students). The exclusions resulted in a final analytic sample of 469 students.

Assignment: The study used a quasi-experimental, instrumental variable design based on both the actual participation in the program and an instrument measuring randomized assignment into the program.

First, the initial randomization to the conditions defined an instrumental variable that was used to adjust the actual program participation for selection bias. Group assignment was based on a lottery (applications submitted at the end of 8^th grade) for admission into the career academy housed within one high school. The authors noted that the majority of students who lost the lottery (92%) enrolled in the regular track (i.e., non-career academy program) at the same high school.

Second, actual participation or non-participation in the program defined the two conditions. Since students who lost the lottery could still enroll in the academy from the waitlist if open seats remained after being offered to lottery winners, the randomization did not completely define condition participation. Both intervention and control group students attended the same school (Apex High School). The intervention group included students who won the lottery and enrolled in the Academy of Information Technology (AOIT), which included a technology-based paid internship, a four-year sequence of IT courses/electives, cohort-based grouping with fellow program enrollees, career development, soft-skills training throughout the four-year program, completion of one of two curricular IT tracks and district-wide elective requirements. As with admission into the academy itself, both track and suggested elective courses based on track were first filled with AOIT students and, if seats remained open, non-AOIT students were then eligible to enroll. The control group included students who lost the lottery and enrolled in the "traditional" high school curriculum track. Neither the number of lottery winners vs. losers nor the number of enrollee compliers vs. non-enrollee compliers were provided.

Assessments/Attrition: The study used administrative data, but attrition was not reported. In the Table A2 table note, the authors stated, "In 2014-15, a second career academy to which students zoned for Apex High School could apply opened in Apex Friendship High School (Academy of Engineering and Advanced Manufacturing) and thus the share of students who lost the AOIT lottery and attended 'regular' Apex High School dropped because many entered the new academy." This statement suggests that there may have been some attrition over the course of the study period.

Sample:

The analytic sample of lottery applicants (n=469) included students who were 36% female, 78.7% non-Hispanic White, 10.4% Asian, 5.1% non-Hispanic Black, and 2.6% Hispanic. Additionally, 8.3% were involved in special education, 57.9% had an academically gifted status, and had 8^th grade standardized academic achievement scores of 0.745 for math and 0.738 for reading.

Measures:

Study measures came from official administrative data housed by the North Carolina Department of Public Instruction, which includes student-level demographic data (i.e., gender, race/ethnicity, special education status, and limited English proficiency [LEP] status), academic engagement and achievement data (e.g., standardized reading and math test scores), and college enrollment data using the National Student Clearinghouse.

The primary outcome was on-time high school graduation (i.e., any time by the end of the fourth year of high school). Secondary behavioral outcomes were attendance, academic performance in high school, and college enrollment. Additional intermediate outcomes were AP course-taking and industry certifications in high school, which represented risk and protective factors.

Analysis:

The researchers conducted two sets of analyses. The first set estimated the effects of winning the lottery on several high school and college outcomes (i.e., intent-to-treat analyses). The second, primary set of analyses used an instrumental variables approach to estimate the average effect of enrolling in the career academy program among students who did so because they were lottery winners on the same outcomes, which is the local average treatment effect (LATE). The instrument was a student's "random" assignment from the lottery, which was used to predict enrollment into the career academy, which was in turn used to estimate the causal effect of the treatment on the outcome variables. The models used two-stage least squares estimation and included pre-lottery covariates (e.g., gender, race/ethnicity, prior academic achievement) and lottery-cohort fixed effects. The authors also conducted a treatment-on-the-treated (TOT) analysis for the attendance outcome.

Exploratory subgroup analyses examined whether program impacts differed by gender and baseline achievement levels (i.e., 8^th grade test scores). Additional intermediate outcome analyses examined whether attendance in 9^th grade, academic performance and AP course-taking during high school, and industry-recognized credential attainment mediated high school completion.

Intent-to-Treat: The first set of analyses that estimated the effects of winning the lottery on outcomes of interest followed an intent-to-treat approach. Although the instrumental variable approach (i.e., the second, primary set of analyses) also included all participants with complete data, it may violate the intent-to-treat criterion by examining the effects of actual participation in the program rather than the effects of random assignment to the program. However, unlike a treatment-of-the treated analysis, the instrumental variable approach minimizes rather than increases selection bias.

Missing Data Method: The analysis included participants with complete data at baseline and follow-up. The authors also conducted sensitivity analyses, described in a footnote on page 168, stating, "Results are nearly identical and conclusions unchanged if we keep these observations in the analytic dataset and control for missing information using indicator variables in our regressions."

Outcomes

Implementation Fidelity: Not reported.

Baseline Equivalence:

The study did not compare the conditions directly but conducted tests for baseline equivalence on the randomized lottery groups. There were no statistically significant differences between students who won the lottery and those who lost the lottery on demographic characteristics or baseline academic achievement outcomes (Table 4).

Differential Attrition:

The study used administrative data, but there were no details about missing student graduation data or other outcome data. However, the authors conducted tests comparing the baseline characteristics of students who won vs. lost the lottery and reported no significant differences (see p. 168 text and Table 4).

Posttest:

Career academy enrollment significantly increased the likelihood of expected, on-time high school graduation by about 8 percentage points. TOT analyses revealed that career academy participation also significantly reduced 9^th grade absences by about 1.4 days. Both of these effects remained significant in ITT analyses that examined differences in program impacts between students who won the lottery and those who lost the lottery. However, academy participation did not have a significant effect on academic performance (reading, math, and composite test scores) or college enrollment.

For the intermediate outcomes, career academy participation significantly increased the likelihood of earning an IT industry certification. Academy participation did not have a significant effect on rates of AP course-taking during high school.

Subgroup analyses showed that the overall effect on high school graduation was driven by impacts on males as well as students in the middle to upper-middle of the baseline math achievement distribution. Participation in the academy increased the likelihood of college enrollment within one year of expected, on-time high school graduation for males but not females.

Long-Term: Not tested, as the program extended through high school.