Early College High School Model

Study 1

Berger et al. (2013, 2014) are full reports written by the American Institutes for Research (AIR) and Haxton et al. (2016) is a peer-reviewed publication. All three articles are the same report, just published in different formats. The writeup for Study 1 refers to Haxton et al. (2016), which is the same as Berger et al. (2013, 2014).

Summary

Haxton et al. (2016) and Song & Zeiser (2019) conducted a multisite randomized controlled trial (using lottery assignments). The study recruited from a five-state sample (i.e., North Carolina, Ohio, South Carolina, Texas, and Utah) of 17 lotteries across 10 schools and 3 cohorts of students entering high school in 2005-06, 2006-07, and 2007-08, which resulted in a sample size of 2,458 students. Haxton et al. (2016) followed participants from the year they entered Grade 9 (i.e., Year 1) to 2 years after expected high school graduation (i.e., Year 4). Song & Zeiser (2019) followed these same participants from the year they entered Grade 9 (i.e., Year 1) to 6 years after expected high school graduation (i.e., Year 10). The primary outcomes included high school graduation rates, as well as college enrollment and completion rates.

Haxton et al. (2016) and Song & Zeiser (2019) found that treatment students were more likely than control to:

• Enroll in college within 2 years and within 6 years of expected high school graduation
• Enroll in 2-year colleges within 2 years and within 6 years of expected high school graduation
• Earn a credential within 2 years and within 6 years of expected high school graduation
• Earn an associates or certificate within 6 years of expected high school graduation
• Earn a bachelor's degree within 6 years of expected high school graduation.

As for risk and protective factors, Haxton et al. (2016) found that, as compared to the control group, treatment students:

• Were more likely to earn college credits in high school
• Report stronger college-going high school cultures and support from instructors while in high school.

Evaluation Methodology

Design:

Recruitment: To be eligible for this retrospective study, an EC (Early College high school) had to meet the following criteria for at least 1 of 3 school years (2005-2006 through 2007-2008): (a) enrolled students in Grades 9 to 12; (b) had high school graduates; (c) were oversubscribed and used lotteries in their admission processes for incoming ninth graders; (d) retained the lottery records; and (e) implemented the EC model as a whole-school program. The potential study sample was restricted to all Early College High Schools that were open by fall 2007 to ensure that students in the study would have had the opportunity to complete at least 2 years of college by the end of data collection (e.g., 2012-2013). Of the 154 Early College High Schools open nationwide by fall 2007, about two-thirds were not eligible for the study because they were undersubscribed. Of the remaining models, 10 met the criteria for inclusion in this study.

Assignment: This study capitalized on retrospective admission lotteries that occurred between 2005-06 and 2007-08. The sample included 17 lotteries across 10 sites and 3 cohorts, with 1,044 students randomly assigned to treatment and 1,414 students randomly assigned to control (total n = 2,458). Treatment students were lottery applicants offered enrollment either through the initial lottery or from a randomized waitlist prior to the first day of school, and control students were lottery applicants who were not offered enrollment. Two schools in the first two cohorts conducted the lottery themselves.

Attrition: Only 2,207 students had high school graduation data (an attrition rate of 10%), and the reasons were not provided (See Table 3; Haxton et al., 2016). The authors reported no attrition (n = 2,458 at both the 2- and 6-year follow-up) for the postsecondary outcomes (i.e., college entrance and completion). In terms of risk & protective factors, Haxton et al. (2016) did not survey students from Cohort 1 because the authors were not confident in students' ability to recall their high school experiences multiple years after leaving high school. The survey response rate was 94% for treatment students and 88% for control students. Non-response-adjusted survey weights were applied to all analyses of survey data.

Sample: The 10 Early College high school were located in five states (North Carolina, Ohio, South Carolina, Texas, and Utah); five schools were located in urban areas, two in midsized cities, and three in small towns. Nine schools opened as new schools, and one was an existing school that became an Early College high school. Eight of the treatment schools had partnerships with 2-year colleges, and the other two had partnerships with 4-year colleges. All 10 schools were small schools (i.e., fewer than 150 students per grade), with an average enrollment of 290 students. Across the 10 schools, 49% of the students were non-White (ranging from 12% to 100%), and 44% were eligible for free or reduced-price lunch (ranging from 9% to 99%). The authors reported student characteristics by condition. Among the treatment group, 51.8% were female; 52.5% were minority; 21.6% were first-generation college going; and 47.3% were low-income students. Among the control group, 52.9% were female; 53.7% were minority; 19.9% were first-generation college going; and 45.2% were low-income.

Measures: The original study (Haxton et al., 2016) gathered administrative data from Early College High Schools, districts, and state departments of education on high school outcomes. The specific data sources for these variables differed by site, and in some sites, they were able to obtain data for the same measure from multiple sources. For high school graduation, the percentage of students earning a high school diploma or general equivalency diploma (GED) was measured.

Data for postsecondary enrollment and degree attainment came from the National Student Clearinghouse (NSC), which collects data from higher education institutions across the country.

• For "college enrollment" and "college degree attainment", whether students enrolled in college or earned any degree by the end of study data collection was examined, including either during or after high school.
• "College degree" meant any postsecondary credential, including certificates, associate's degrees, or bachelor's degrees.

For the original article (Haxton et al., 2016), college enrollment and degree attainment outcomes were measured in three ways:

1. Outcomes at any point in the study period;
2. Cumulative outcomes by Year 4, Year 5, and Year 6; and
3. Outcomes after high school (after Year 4).

Year 4 reflected the period when students would traditionally be in their final year of high school. For students on a traditional trajectory, Year 5 was the year immediately following high school graduation. Year 6 (i.e., 2 years after high school graduation in a typical time frame) was the last year for which the authors had data for all students in the study. The study analyzed college outcomes at different points in time to parse out the timing of the effects, and thus examine whether the treatment impact was concentrated in the high school years or whether the impact persisted after high school.

Both articles (Haxton et al., 2016; Song & Zeiser, 2019) divided postsecondary outcomes into two domains with three primary outcomes in each domain:

1. Outcomes in the college enrollment domain.
   1. Enrolled in college within 2 years (Haxton et al., 2016) and within 6 years (Song & Zeiser, 2019) after expected high school graduation.
   2. Enrolled in a 2-year college within 2 years (Haxton et al., 2016) and within 6 years (Song & Zeiser, 2019) after expected high school graduation.
   3. Enrolled in a 4-year college within 2 years (Haxton et al., 2016) and within 6 years (Song & Zeiser, 2019) after expected high school graduation. Note - Song & Zeiser (2019) also examined enrollment in a selective 4-year college using information from Barron's Profiles of American Colleges.
2. Outcomes in the degree attainment domain.
   1. Completed any postsecondary degree within 2 years (Haxton et al., 2016) and within 6 years (Song & Zeiser, 2019) after expected high school graduation.
   2. Completed an associate's degree or certificate within 2 years (Haxton et al., 2016) and within 6 years (Song & Zeiser, 2019) after expected high school graduation.
   3. Completed a bachelor's degree within 2 years (Haxton et al., 2016) and within 6 years (Song & Zeiser, 2019) after expected high school graduation.

Haxton et al. (2016) also collected data on students' high school experiences through a student survey. The survey was administered in 2012-after their expected high school graduation-to 1,416 randomly selected students in the two oldest cohorts. Students' experiences during high school were measured through college exposure and student supports. For college exposure, students' college course-taking and credit accumulation as well as their Advanced Placement (AP) course-taking and exam passage in high school were measured. For student supports, the college-going culture in the school (a 1 to 4 scale based on three survey items, α reliability = .80), instructor support (a 1 to 4 scale based on six items, α reliability = .88) and whether students had access to general information about college were measured.

Analysis:

To estimate the overall program effects across lotteries, authors constructed a two-level model that took into account the clustering of students within lotteries. The treatment indicator was group-mean centered at the student level to make sure the comparisons of treatment students and control students were made within, rather than across, lotteries and thus produced unbiased estimates. The authors modeled the intercept as a random effect to take into account the clustering of student outcomes within lotteries and modeled the treatment effect as fixed at the lottery level because the number of lotteries in the study was too small to generate stable estimates of the variation in treatment effects across lotteries. Across all analyses, all models included the following student background characteristics as covariates: gender, race and ethnicity, first-generation college-going status, low-income status, and academic achievement prior to high school.

To guard against the risk for inflated Type I error due to multiple comparisons, Song & Zeiser (2019) designated a single outcome in each outcome domain (i.e., ever enrolled in college and ever completed a postsecondary degree or certificate) as the "confirmatory" outcome, and their main conclusion about the treatment effect in each domain is based on the confirmatory outcome. To check the robustness of findings to potentially increased Type I error due to multiple comparisons, the authors also used the Benjamini-Hochberg method to correct for multiple comparisons within each outcome domain. Because there are three primary outcome measures within each outcome domain, Song & Zeiser (2019) corrected for three comparisons within each domain. There is, however, no mention of confirmatory outcomes or of corrections for multiple comparisons in Haxton et al. (2016).

In addition to main effects, Haxton et al. (2016) also conducted moderation analyses on student subgroups (e.g., gender, minority status, first generation college-going status, low-income status, or level of prior mathematics and ELA achievement) for the three key outcomes: 1) high school graduation, 2) college enrollment, and 3) college degree attainment. Moderation analyses in Song & Zeiser (2019) examined the impact on college enrollment and completion of treatment interacted with gender, race/ethnicity, income status and level of prior mathematics and ELA achievement.

Intent-to-Treat: All participants were analyzed according to the condition in which they were assigned and all available data were utilized in the analysis which is in line with intent-to-treat protocol. The primary impact analyses used multiple imputation for missing covariates, and included all background and outcome variables available from both the extant data and the student survey data. Researchers generated 10 imputed data sets, conducted all analyses using each imputed data set separately, and then combined estimates across the 10 data sets, taking into account the uncertainty in imputed values both within and across the imputed data sets. Administrative records assessing postsecondary outcomes came from the National Student Clearinghouse (NSC), which covers more than 98% of all student enrollments in public and private colleges and universities. The authors assumed that students for whom the NSC could not find matching records did not attend college or attain a postsecondary degree; however students with missing outcome data were excluded from the analysis.

Outcomes

Implementation Fidelity: Not reported.

Baseline Equivalence: Baseline equivalence was tested on a variety of background characteristics including gender, minority status (i.e., non-White), eligibility for free or reduced-price lunch, first-generation status, and English language arts (ELA) and mathematics test scores prior to high school, and there was a significant difference in English language arts scores in favor of treatment. This difference was not reported as significant (reported as p = .068) in Song and Zeiser (2019). It may be the case that this was caused by recalculating imputed data (which they did for missing covariate data), but it is ultimately unclear what caused this inconsistency.

Differential Attrition: There was high attrition (10%) for high school graduation rates (Haxton et al., 2016), but tests for differential attrition were not conducted. However, the authors reported no attrition on the postsecondary outcome measures measured in Haxton et al. (2016) and Song & Zeiser (2019).

Posttest: Haxton et al. (2016) reported no significant difference between treatment and control in high school graduation rates.

As for risk and protective factors, Haxton et al. (2016) found that, as compared to the control group, treatment students were more likely to:

• earn "any" college credits (OR = 8.04) and 1 year of college credit (OR = 18.61) in high school,
• report stronger college-going cultures (ES = .32) and support from instructors (ES = .32) in their high schools.
• have access to general college information in school than the control group (OR = 1.31), though this effect was marginally significant (p = .06). I

In contrast, students in the treatment group, compared to the control group, were less likely to take at least one Advanced Placement (AP) course (OR = 0.23) and pass at least one AP exam (OR = 0.19).

Long-Term: Haxton et al. (2016) reported positive long-term impacts for the overall sample on three of six outcomes. In terms of college access, within two years of expected high school graduation, treatment students were more likely to:

1. Enroll in college
   1. OR = 1.63
   2. Marginal effects: 80.9% of treatment students had at least one record of college enrollment, whereas roughly 9 percentage points higher than the 72.2% college enrollment rate for control students.
2. Enroll in a 2-year college
   1. OR = 2.33
   2. Marginal effects: 60.8% of treatment students had enrolled in a 2-year college compared to 40% of control students.
3. Attain a postsecondary degree
   1. OR = 6.71
   2. Marginal effects: 24.9% of treatment students earned postsecondary degrees compared with 4.7% of control students.

Haxton et al. (2016) also found impacts in favor of treatment were detected for college enrollment by Year 4 (OR = 5.41), Year 5 (OR = 1.72), and Year 6 (OR = 1.73).

When examining enrollment by school type (2 vs. 4-year colleges), compared to the control group, students in the treatment group were more likely to enroll in a:

• 2-year college by Year 4 (OR = 6.82), Year 5 (OR = 2.85), and Year 6 (OR = 2.46)
• 4-year college by Year 4 (OR = 2.92) and Year 6 (OR = 1.32).

In terms of college attainment, compared to control, students in the treatment group were more likely to have earned:  

• "any degree" by Year 4 (OR = 35.37), Year 5 (OR = 21.38) and Year 6 (OR = 14.66).
• an Associate's degree (OR = 12.14) or a Bachelor's degree by (OR = 3.80) by Year 4

Song & Zeiser (2019) reported significant, positive effects on five of six outcomes measured six years after expected high school graduation. Note: Song & Zeiser (2019) computed effect sizes for binary outcomes by dividing the logged odds ratio of each outcome (reported in Appendix F, p. 65 and p. 68) by 1.65 (i.e., the Cox index). Each year between the fourth year of high school and 6 years after expected high school graduation, findings show treatment students were more likely than control to:

1. Enroll in college
   1. ES = .281
   2. Marginal effects: 84.2% of treatment students enrolled in college, roughly 7 percentage points higher than the college enrollment rate of 77% for control students
2. Enroll in 2-year colleges
   1. ES = .474
   2. Marginal effects: 65.8% for treatment students compared to 46.8% for control students
3. Earn a credential
   1. ES = .305
   2. Marginal effects: 45.4% of treatment students completed postsecondary degrees, compared with 33.5% of control students
4. Earn an associates or certificate
   1. ES = .725
   2. Marginal effects: 29.3% of treatment students compared with 11.1% of control students)
5. Earn a bachelor's degree
   1. ES = .157
   2. Marginal effects: 30.1% of treatment students compared with 24.9% of control students) each year between the fourth year of high school and 6 years after expected high school graduation

Note: the authors re-estimated treatment impacts after removing the one treatment school where student demographic and prior achievement data had to be imputed for all treatment and control students because their state did not allow researchers to link student background data to outcome data. Results from this set of analyses mirror the ITT results presented in the main text with one exception: the treatment impact on bachelor's degree completion within 6 years after expected high school graduation (i.e., by Year 10) was slightly smaller and only marginally significant (p = .096) after removing one site where background data were imputed for all students.

By the end of the sixth year after expected high school graduation, the two groups of students did not significantly differ in the rate of enrolling in

• 4-year colleges (57.6% of EC students compared with 56.7% of control students)
• Selective 4-year colleges (by the end of Year 10, approximately 18% of both treatment and control students had enrolled in selective 4-year colleges).

Regarding the impact over time,

• For enrollment in any type of postsecondary institutions, treatment had a significant impact on enrollment rates by the end of each year between Year 4 (i.e., the fourth year of high school) and Year 10 (i.e., 6 years after expected high school graduation), although the size of the impact tended to decrease over time (Song & Zeiser, 2019).
• This finding was consistent with enrollment in 2-year colleges.
• Similarly, while the difference in degree completion rate between the treatment and control students remained statistically significant over time, the size of the difference decreased from a maximum of 22.5 percentage points (by the end of Year 6 and Year 7) to 11.9 percentage points (by the end of Year 10).
• However, regarding the completion of an associate's degree or certificate, the results show that the differences between treatment and control students were largely stable in both size and statistical significance over time.
• Where bachelor's degree completion is concerned, the difference between the two study groups first widened and then narrowed (but remained statistically significant) over the period examined.

In terms of subgroup analyses,

• The program effect on college degree attainment (in favor of the treatment group) within two years of expected high school graduation was significantly stronger for minority students and low-income students, and students who entered high school with higher math and ELA scores (Haxton et al., 2016).
• Song & Zeiser (2019) found no subgroup effects in terms of family background characteristics within 6 years after expected high school graduation.
• Treatment impacts on enrollment in 2-year colleges and completion of an associate's degree or certificate within 6 years after expected high school graduation (i.e., by Year 10) were stronger for students with higher levels of Grade 8 achievement (Song & Zeiser, 2019).

Study 2

Summary

Edmunds et al. (2012, 2017, 2020) conducted a multisite randomized controlled trial (using lottery assignments). The study included students who applied to one of 12 early college schools in North Carolina and included 18 cohorts of students who enrolled in ninth grade in the 2005-06, 2006-07, 2007-08 and 2008-09 school years (n =1,689). Edmunds et al. (2020) included two more cohorts for the overall study but the sample size varied by outcome; thus, the full study examined 4,054 students who applied to 19 urban and rural early colleges in North Carolina over a series of 6 years. The first cohort for the Edmunds et al. (2020) study was in ninth grade in 2005-2006 and the final cohort was in ninth grade in 2010-2011. The study followed participants up to 6 years after students completed 9^th grade, which is 2 years after high school graduation in a typical time frame (Edmunds et al., 2012, 2017). Long-term outcomes for Study 2 were examined at two time points: 4 years after completion of 12th grade and 6 years after completion of 12th grade (Edmunds et al., 2020). The primary outcomes included high school graduation rates, as well as college enrollment and completion rates.

Edmunds et al. (2017, 2020) found that treatment students as compared to the control students were more likely to have:

• Enrolled in postsecondary education (2 years after high school graduation in a typical time frame)
• Attained a postsecondary degree 2 years after high school graduation in a typical time frame and 6 years after the end of Grade 12
• Attained an associate degree 6 years after the end of Grade 12.

In terms of risk & protective factors, compared to the control group, students in the treatment group:

• Took more core college preparatory courses and succeeded in them at the end of 9^th grade.
• Had better school attendance and fewer suspensions at the end of 9^th grade.
• Earned more college credits by the end of 12^th grade.

Evaluation Methodology

Design:

Recruitment: The schools recruited for this study were in rural and urban settings in all regions of North Carolina. The authors reported that while demographically similar to traditional schools, the early colleges included in this study were smaller than the traditional schools in their counties, had much lower enrollments of students with disabilities, and enrolled students with higher initial levels of achievement. To participate in the study, schools had to have more applicants than available slots and had to agree to use a lottery to randomly assign students. Schools entered the study on a rolling basis, and if they continued to use the lottery, they could continue to contribute cohorts of students to the study.

Assignment: Edmunds et al. (2017) reported a total of 1,689 students who applied to 12 different early colleges across North Carolina and enrolled in ninth grade in the 2005-2006, 2006-2007, 2007-2008, and 2008-2009 school years were enrolled in the lottery (though Edmunds et al., 2012, reported 1,607 students were in the lottery, and Edmunds et al., 2020 reported 1,687 students in the lottery, so the exact sample size is unclear). These 12 schools enrolled a total of 18 cohorts of students, with 5 schools enrolling multiple cohorts. Beginning with the 2007-2008 cohort, the research team began conducting the lotteries (though 2 schools in the first cohort conducted their own lottery). Lottery assignments operated as follows. Each applicant who met the school's eligibility criteria was assigned a random number and the list of students was ordered from lowest to highest, with the lowest numbers being selected into the early college until all available slots were filled. Edmunds et al. (2020) noted on page 262: "Some schools requested that lotteries be further stratified by selected student demographic characteristics to accommodate their specific priorities…. The sample for each early college was thus a function of the number of eligible applicants each school had, the number of slots they were trying to fill, and the extent to which any additional stratification reduced the number of students randomized (which might have happened if all students in a specific stratum were accepted and therefore had to be excluded from the study)." As a result of the lottery process for randomization, 953 students were assigned to treatment and 736 were assigned to the control condition.

In addition, the sample used for the postsecondary GPA analysis reported in Edmunds et al. (2020) included students who applied to the 19 study early colleges from 2005-2006 through 2010-2011 and enrolled in a University of North Carolina (UNC) campus post high school through the spring 2017 semester. Since the GPA dataset was limited only to students who enrolled in a UNC school, a quasi-experimental analysis was conducted to evaluate GPA outcomes and authors employed propensity score weighting methods separately for each of the 4 GPA measures to create comparable treatment and control groups. The first step was estimation of the propensity scores (or the probability of having a GPA as a function of baseline covariates that are considered to predict GPA and enrolling in a UNC campus). Covariates included demographics (race/ethnicity, gender, age, economic disadvantage, first-generation college going status, having a disability, being identified as academically or intellectually gifted), baseline indicators of student achievement (being retained in a prior grade, scores in eighth-grade math and reading end-of-course exams, passing Algebra I in eighth grade, and teachers' assessment of eighth-grade achievement in math and reading), eighth-grade absences (which is considered a proxy for academic engagement and motivation), and additional factors that were expected to predict enrolling in UNC such as academic performance of the eighth-grade middle schools, district-level baseline high school graduation rates, and number of colleges in the eighth-grade county. Next, weights for the treatment students with valid GPA measures were calculated so they looked similar to control students with valid GPA measures. This process resulted in an analytic sample across the four GPA measures that varied from 1,072 students (674 treatment and 398 control) for cumulative GPA through second year of college to 1,292 students (797 treatment and 495 control) GPA through first year of college.

Attrition: Using 1,607 as the randomized sample, Edmunds et al. (2012) reported attrition rates of 18% (1315/1607) for algebra, and less than 1% for college prep math courses (1607/1607), college prep English courses (1607/1607), and absences/suspensions/plans to attend a 4-year college (1604/1607). Meanwhile, Edmunds et al. (2017) reported, "Because we use extant administrative data, we are able to include almost all students from the original lottery samples in our analyses" (p. 304). Overall attrition rates, however, varied according to the outcome measures. For enrollment in postsecondary education and attainment of a postsecondary credential, overall attrition rates were 2% (1651/1689). For high school graduation, it was 6% (1594/1689), and for college credits attained in high school, it was 15% (1437/1689). In Edmunds et al. (2020), there was "virtually no overall or differential attrition" (p. 262) for credential outcomes since students with no record in the National Student Clearinghouse database were coded as a zero (i.e., no credential; see further explanation under "intent to treat" section). In addition, while the sample sizes for the four different versions of GPA outcomes varied slightly because the sample was selected based on whether a valid GPA measure existed in the dataset, there was no attrition in the samples assessing postsecondary GPA outcomes (see Table 3, p. 268).  

Sample: The overall sample for the lottery sample reported in Edmunds et al. (2012, 2017, 2020) was 41% male, and the racial/ethnic composition of the students was 60.2% white, 26.7% black, and 8.3% Hispanic. Forty-one percent of the students were first-generation college going and 50.7% were eligible for free or reduced lunch. Participating schools (n=12) served about 142 students (on average); and had a teacher turnover rate of 17% and a novice teacher rate of 31.2%. Edmunds et al. (2020) do not report the demographics for the GPA samples (just that the samples did not differ by condition after the matching process).

Measures:

Edmunds (2012) used administrative data to measure college-prep math courses taken in 9^th grade, defined as taking/passing Algebra I, Geometry, or Algebra II. Administrative records were also used to measure 9^th grade school attendance and the percentage of students who had been suspended at least once in 9^th grade. Students' plans to attend a 4-year college were assessed using data from a survey that accompanied each state-mandated end-of-course exam during the period of the study.

Edmunds et al. (2017) used administrative data collected by 3 primary sources: 1) the North Carolina Department of Public Instruction, 2) the National Student Clearinghouse, and 3) the North Carolina Community College System.

• College credits earned while in high school: The number of college credits transferable to a 4-year college earned though the end of 12^th grade was measured. Vocational courses and remedial or developmental courses were excluded as these credits were not transferable to a 4-year college, and "passing" was defined as earning a C or above. Data came from the North Carolina Community College System, and thus excluded 36 students who were enrolled in 4-year colleges while in high school. As for AP exam, data were not available before 2009-2010. However, the authors report sensitivity analyses that excluded 248 students who would have been in 11th and 12th grades (the years in which most students take AP classes) prior to 2009-2010 showed very little difference.
• Graduation from high school: The authors reported 5-year graduation rates (for both treatment and control) because the majority of early colleges are 5 year programs.
• Enrollment in postsecondary education: The authors reported whether a student was ever enrolled in any type of postsecondary education (part-time or full-time). This enrollment could have occurred at any point over the time period from 9^th grade through the fall semester of the 6^th year after the student started high school.
• Postsecondary credentials: Any type of postsecondary credential, including associate degree, technical credential, or bachelor's degree, was assessed. The sample only went through the sixth year after students' entrance into high school (or two years after the student's expected graduation from high school), allowing two years for students in two-year institutions to complete their degrees.

Edmunds et al. (2020) used administrative data and focused on two long-term outcomes: (1) attainment of a postsecondary credential, as measured by the National Student Clearinghouse database; and (2) postsecondary GPA, as measured by administrative data collected through the University of North Carolina (UNC) system. Covariates were collected from the North Carolina Department of Public Instruction.

• Postsecondary credentials: Any type of postsecondary credential, including associate degree, technical credential, or bachelor's degree, was assessed. Results were presented both for overall attainment of any credential and separately for each degree type, and findings were presented both 4 and 6 years after completion of 12^th grade (allowing for 4-6 years for students to complete their degrees).
• Postsecondary GPA: Cumulative GPA was assessed for all college courses students took after entering the University of North Carolina (UNC) system after graduating from an early college. GPA was measured at four time points (a) through 2 years after 12th grade, (b) through 3 years after 12th grade, (c) through students' first year at the UNC system, and (d) through students' second year at the UNC system. Cumulative GPA measured at 2 and 3 years after 12th grade aim to hold constant students' age and the time after they enrolled in high school, whereas cumulative GPA through the first and second year at the UNC system aim to hold constant the time students spent in the UNC system.

Analysis: Edmunds et al. (2012) conducted an ITT analysis. Primary impact estimates were obtained from a multivariate linear regression model, with a fixed Treatment X Block (school-level) interaction term. Authors calculated an overall ITT impact estimate by averaging these block-specific effects, weighting them proportionally to the total number of students (treatment and control) in each block. In addition, several sensitivity and specification tests were run (none of which yielded results substantively different from those yielded by the primary analytic strategy). Authors adjusted for multiple comparisons for all core outcomes assessing main effects using the Benjamini-Hochberg multiple comparisons correction.

Edmunds et al. (2017, 2020) used multivariate linear regression models that included lottery indicators (or lottery fixed effects), a treatment indicator capturing the initial group to which a student was randomly assigned, and baseline student characteristics including demographic characteristics such as gender, race/ethnicity, age, free/reduced price lunch status, special education status, whether a student was retained prior to eighth grade, and eighth-grade academic performance. The statistical inference took into account clustering of students within schools by calculating cluster-robust standard errors. The models were estimated using weights that were based on students' probability of being selected into the early college. This was done because some schools conducted a stratified lottery, which led to different probabilities of selection into the treatment condition.

The propensity score analysis conducted for the GPA measures reported in Edmunds et al. (2020) utilized the same covariates as those used with the randomized sample reported in the article, and also included additional measures such as eighth-grade absences, teachers' assessment of students reading and math achievement in eighth grade, performance score for the eighth-grade school, district-level average high school graduation rates, and number of colleges within eighth grade county. Similar models were run on sub-group analyses, which examine results by race/ethnicity, income, achievement level (i.e., entered high school below grade level), and first-generation status.

In addition to the ITT analysis, Edmunds et al. (2012) presented instrumental variable estimates to represent the average effect of attending an early college (local average treatment effect [LATE]).

Intent-to-Treat: All participants were analyzed according to the condition in which they were assigned, all available data were utilized in the analysis, and authors stated they conducted an ITT analysis. Students that did not have a record in the National Student Clearinghouse database were considered to not have earned a credential in Edmunds et al. (2020) even though a student could be missing from the database for a variety of reasons beyond nonenrollment or nondegree attainment. This approach ensured outcome data for all randomized students with outcomes defined in the same way for both treatment and control groups. To address missing covariate values, Edmunds et al. (2020) utilized a multiple stochastic imputation method consistent with What Works Clearinghouse 4.1 standards. 

Outcomes

Implementation Fidelity: Not reported.

Baseline Equivalence: Out of 13 baseline variables measured for the randomized sample reported in Edmunds et al. (2012, 2017, 2020), including 9 socio-demographic (Black, Hispanic, White; gender; first-generation college, free/reduced price lunch eligibility; disabled/impaired, gifted, retained) and 4 pretest scores (8^th grade reading and math scores; percent passed reading and math in 8^th grade), 2 significant differences were detected between treatment and control at baseline - being retained in elementary or middle school (effect size = 0.37) and passing the eighth-grade math exam (effect size = 0.16), both in favor of the treatment group. For the quasi-experimental analytic sample assessing GPA outcomes reported in Edmunds et al. (2020), weighting reduced all 23 baseline covariates to differences below 0.10 SDs.

Differential Attrition: Not conducted. Attrition was >5% for 1 (college credits received) of the 3 outcomes reported in Edmunds et al. (2012, 2017); otherwise attrition was low for Edmunds et al. (2012, 2018, 2020).

Posttest: In the ITT analysis, Edmunds et al. (2012) found that compared to the control group, students in the treatment group had significantly higher school attendance and lower suspension rates at the end of 9^th grade. Though the LATE estimates were similar and presented in Table 3, no significance levels were reported. Edmunds et al. (2017) found that treatment students earned significantly more college credits than control students while in high school. That is, by the end of 12th grade, treatment students had earned an average of 21.6 transferable college credits compared to an average of 2.8 credits earned by the control group. Edmunds et al. (2017) reported no impact on five-year high school graduation rates.

Long-Term: At the posttest (2 years after high school graduation in a typical time frame), Edmunds et al. (2017) found that treatment students were more likely to be enrolled in postsecondary education than control students. Specifically, by the beginning of the 6^th year after entering 9^th grade, 89.9% of the treatment group had enrolled in postsecondary education at least once (including enrollment while in high school), compared to 74.3% of the control group. In addition, more treatment students attained some sort of postsecondary credential compared to the control group; 30.1% of treatment group students, compared to 4.2% of the control group, had attained a postsecondary credential.

Results reported in Edmunds et al. (2020) showed that treatment students received postsecondary credentials at a higher rate than control students: By the end of the fourth year after the end of Grade 12, 37.8% of the treatment group had earned a postsecondary degree compared with 22.0% of the control group. The authors note, however, that is result was largely driven by a 21.2 percentage point impact on associate degree attainment. In looking 6 years after the end of Grade 12, findings showed a significant impact on overall degree attainment and on associate degree attainment with 44.3% of the treatment group estimated to have a postsecondary credential compared with 33.0% of the control group and an impact of 21.8 percentage points on associate degree attainment. However, by 6 years after Grade 12, the control students have essentially caught up to the treatment students in 4-year degree attainment (24.9% treatment vs. 24.0% control).

Edmunds et al. (2020) showed no impact on postsecondary GPA.

Subgroup results showed large positive impacts for associate degree but non-significant impacts for on 4-year degrees with one exception: There is a statistically significant positive impact on 4-year degree attainment for economically-disadvantaged students.

Edmunds et al. (2020) also found that a larger number of treatment students that had obtained 2- and 4-year degrees obtained their degrees at a faster pace than control students who obtained 2- and 4-year degrees. Analyses found that treatment students who earned an associate degree did so approximately 2 years earlier than the control students. Treatment students who earned a bachelor's degree did so approximately half a year earlier than control students.

Study 3

The program was adapted for English learners but maintained the key program structure. Those participating in the program during 12th grade had their high school classes in the morning and then were bused to college classes at a nearby community college in the afternoon.

Summary

Johnson and Mercado-Garcia (2022) used a difference-in-differences quasi-experimental design to examine 15,090 English-learner students in 17 California high schools. Cohorts of students in three high schools that graduated both before and after program implementation were compared to cohorts of students in 14 high schools that did not implement the program. Outcomes included on-time high school graduation and college enrollment.

Johnson and Mercado-Garcia (2022) found that English-learner students attending treatment high schools during the post-implementation period, relative to English-learner students attending control high schools during the post-implementation period, had significantly greater

• On-time high-school graduation
• College credits earned during high school (a risk and protective factor).

Evaluation Methodology

Design:

Recruitment: The sample came from 17 high schools in a single school district in California with a large English-learner population (i.e., 12-20% with limited English proficiency). Students included in the study had been or still were English learners and were part of seven cohorts that graduated in the years 2013-2019. The schools contained 26,311 students total, but the main analytic sample consisted of 15,090 English-learner students.

Assignment: The study compared three high schools that implemented the program with 14 high schools that did not adopt Early College. The difference-in-differences design further divided the 15,090 students into four groups:

• those graduating from the three treatment high schools before program implementation, from 2012/2013 to 2015/2016 (n = 989),
• those graduating from the 14 control high schools before the years of program implementation, from 2012/2013 to 2015/2016 (n = 8,057)
• those graduating from the three treatment high schools after program implementation, from 2016/2017 to 2018/2019 (n = 369),
• those graduating from the 14 control high schools after the years of program implementation, from 2016/2017 to 2018/2019 (n = 5,675).

The design did not follow individual students before and after the program. Rather, it compared cohorts of different students graduating before and after the program. Schools self-selected into the treatment and control groups, but the comparison of student cohorts both before and after implementation minimizes selection bias. The design requires parallel (though not identical) trends for the treatment and control schools in the pre-implementation period and expects divergence of the trends in the post-period.

Among the three treatment high schools, one implemented the program in spring 2017, one in fall 2017, and the third in fall 2018. Of the seven cohorts included in the study, four graduated before implementation and defined the pre-implementation period (see Figure 1).

Assessments/Attrition: Student cohorts of English learners were examined at the end of high school and into the fall following high school (shortly after the end of the 12th-grade program). Because the design did not follow students over time, but instead used all available data on different cohorts before and after implementation, there was no attrition.

Sample:

About 47% of the English-learner sample was female. Slightly more than half identified as ethnically Chinese and 28% as Hispanic. About a third of the students in the sample had parents who reported graduating from high school.

Measures:

Data on students in all seven cohorts came from district high school administrative records and from information on college enrollment held at the National Student Clearinghouse. One risk and protective factor, college credits earned in 12th grade, related closely to program participation. Two other measures captured the two primary behavioral outcomes: immediate enrollment in any certificate, two-year, or four-year college in the fall following the cohort's high school graduation, and enrollment in a four-year college. A third behavioral outcome, on-time high school graduation, served as a secondary measure according to the authors, as the 12th-grade program began too late to substantially affect graduation. The 15% of students without a high school graduation code were assumed to have not graduated on time or attended college.

Analysis:

The difference-in differences analysis used ordinary least squares models for continuous outcomes and linear probability models for binary outcomes. The models included fixed effects for high schools and robust errors clustered at the cohort-school level. A time measure included both pre-implementation (or baseline) and post-implementation periods. Controls included multiple student covariates. The key test of the program came from coefficients representing the combination of attending a treatment school during the post-implementation period.

In addition to the difference-in-differences analyses, the authors presented a synthetic cohort difference-in-differences analysis that used weights to match the treatment and control high schools and a difference-in-difference-in-differences analysis that included non-English learners in the schools (n = 26,311).

Intent-to-Treat: The analysis met the intent-to-treat criterion by using all students eligible for the program in the treatment schools during the post-implementation period rather than those who actually enrolled or participated in the dual program.

Outcomes

Implementation Fidelity:

Not reported, though program effects on college credits (see below) may be seen as a measure of program participation.

Baseline Equivalence:

The design did not require condition equivalence in the pre-implementation period but instead required parallel trends in the outcomes across conditions during the pre-implementation period. Plots for the treatment and control schools in Figure 1 show outcome trends that appear to be parallel.

However, statistical tests of the assumption in Table 5 indicated some possible violations for immediate college enrollment and on-time high school graduation three years before program implementation.

Differential Attrition:

There was no attrition.

Posttest:

The difference-in-differences models in Table 3 showed that English-learner students attending treatment high schools during the post-implementation period had significantly more college credits earned in high school (a risk and protective factor) and significantly higher on-time high-school graduation (a behavioral outcome) than English-learner students attending control high schools during the post-implementation period. However, the models also showed a significant negative or harmful effect on four-year college attendance and no effect on immediate college enrollment.

The synthetic cohort estimates in Table 4 replicated previous results, except that the harmful effect on four-year college enrollment was no longer significant. The difference-in-difference-in-differences estimates in Table 6 showed no significant effects (p < .05) on any of the four outcomes.

Long-Term:

Not examined.