CUNY Start

Study 1

Scrivener et al. (2018) report results only for the semester in which the program is being delivered. Unless otherwise mentioned, the description of the study refers to the full period of evaluation examined in Weiss et al. (2021)

Evaluation Methodology

Design:

Recruitment: The sample came from four large CUNY colleges in New York City that had at least two years experience with the program and sufficient program infrastructure to continue its operation. Eligible student participants had remedial educational needs for college and volunteered for the program. A total of 4,434 students applied, and 3,873 students agreed to participate and could be randomly assigned (excluding the 13% of applicants who, for a variety of reasons, were guaranteed a spot in program). Three cohorts of students took part in the study, starting in spring 2015, fall 2015, or spring 2016 semesters.

Assignment: The study randomly assigned the 3,873 students to the program or a usual-services control group within blocks defined by cohort, intended college, and intended program (full- or part-time). Assignment was done in real time via a web portal controlled by the research team. The probability of being assigned to the program group was 78%, on average. The uneven random assignment ratio was, by design, chosen to help staff fill all program slots. After assignment, 38 participants were dropped (0.8% of program group students and 1.6% of control group students) because they asked to withdraw or did not submit signed informed consent forms. The analytic sample thus consisted of 3,835 students, including 2,997 program group students and 838 control group students.

However, the authors emphasized that CUNY had another program, Accelerated Study in Associate Programs, or ASAP (a model plus Blueprints program), that made it hard to isolate the effects of CUNY Start and may be seen as both a confound and a mediator.

Assessments/Attrition: Data were available for up to three years or six semesters, from spring 2015 (the program semester for the first cohort) through winter 2019 (three years after the third cohort's program semester). After the end of the one semester program, the follow-up period lasted three years. The analysis sample of 3,835 students meant a completion rate of 99%. However, Table A-2 shows that up to 35% of the sample had missing data on baseline measures.

Sample:

The analytic sample included more women than men and was divided almost equally between students aged 19 or younger and students aged 20 or older. Black (26%) and Hispanic (36%) students made up most of the sample, and a substantial proportion (37%) of the sample had a native language other than English. The proportion of students in the sample who were not the first in their family to attend college was almost double the proportion of students who were.

Measures:

The outcome measures came from administrative records of CUNY and the National Student Clearinghouse. The three confirmatory outcomes were measured for the full period and, when appropriate, for each semester: 1) college readiness as the number of college-ready subject areas based on standardized test scores or passing courses (available only for CUNY students and missing for about 3.7% of the sample), 2) academic progress as the cumulative college-level credits earned (available only for CUNY students), and 3) completion of a degree or certificate credential (available for all students from CUNY or the National Student Clearinghouse). Other measures included enrollment, developmental and college-level courses passed, and completion of gateway courses. The single-item measures lacked details on reliability and validity but appear suitable for the study and were independently gathered.

Analysis:

The analysis used regression models with controls for assignment blocks, baseline measures, and missing data indicators. Significance tests used robust (Huber-White) standard errors.

Intent-to-Treat: The analysis included nearly all participants and examined the effect of receiving a program offer rather than participating in the program.

Outcomes

Implementation Fidelity:

The authors reported without quantitative figures that the program was implemented with fidelity. Around 81% of program group members participated in CUNY Start and less than 1% of the control group participated.

Baseline Equivalence:

The tests for baseline equivalence used the analysis sample, which differed from the randomized sample by only 1%. There were no significant differences between program and control group members on any of the 13 baseline characteristics shown in Table 1 or any of the 17 baseline characteristics shown in Table A-2. Furthermore, an omnibus F-test showed that students' baseline characteristics were not jointly predictive of the condition (p = .885).

Differential Attrition:

Attrition was minimal (1%). Data were missing for baseline measures but, given nearly complete data for the outcomes, did not affect the final sample size.

Posttest and Long-Term:

Preliminary results at the end of the program semester reported in Scrivener et al. (2018) showed that the intervention group had passed significantly more college-ready subject areas and had higher CUNY assessment test scores than the control group.

The key results in Weiss et al. (2021) covering six semesters span both the posttest and long-term. The confirmatory results in Table 3 indicated significant program effects on two of three outcomes. The intervention group had significantly more college-ready subject areas and rates of certificate completion (14.5% versus 11.4%). Cumulative credits earned did not differ across conditions. Results presented separately by semester showed that the intervention group had significantly greater enrollment rates in the first three semesters but not the last three. The number of college-ready subject areas differed significantly in all six semesters, while credential completion differed only in semester six. Subgroup analyses suggested that the program most benefitted triple-remedial students but gave little evidence of any other moderation.

However, the authors stated that the one-semester CUNY Start model by itself did not appear to "meaningfully increase college-level credit accumulation or graduation rates in the absence of ASAP or something similar." In support of this point, a mediation analysis found that participation in ASAP accounted for most of the effect of CUNY Start on college-ready subject areas and credential completion. The authors note that CUNY Start enabled more treatment group students to become eligible for and participate in CUNY's Accelerated Study in Associate Programs (ASAP), a separate and highly effective three-year program. Since the control group had lower participation in ASAP, the effects of the two programs appear to be confounded.

Study 2

Summary

Webber (2018) conducted a quasi-experimental study of 11,198 entering students at CUNY. Students who self-selected into the CUNY Start Program (n = 5,599) were matched and compared with the same number of students who did not participate in the program. Outcomes measured at one year and two years after entrance focused on passing remedial courses in English and math.

Webber (2018) found that, relative to the matched comparison group, the CUNY Start students passed significantly more

• English and math remedial gateway courses at posttest and long-term.

Evaluation Methodology

Design:

Recruitment: The study included a sample of students who entered either CUNY Start or a CUNY college as first-time freshmen with at least one remedial need between Fall 2013 and Spring 2015 (total n = 45,594). The students were required on entrance to take gateway courses to fulfill remediation requirements.

Assignment: The first-time students self-selected into the CUNY Start intervention group (n = 5,760) or to regular enrollment (n = 39,834). The intervention group postponed regular coursework to complete the one-semester program, while others began regular coursework immediately.

The study used propensity score matching to minimize condition differences in observed measures. The propensity score matching used a one-to-one matching algorithm without replacement (i.e., a "greedy" match) within cohort and college. The caliper restriction was tweaked by school and semester in order to ensure balance and maximize sample retention, and it ranged from 0.05 to 0.1. Propensity scores were based on demographics (gender, race and ethnicity, age, native language, country of origin, borough of residence) and academic background (initial remedial status, placement exam scores).

The final matched sample of 11,198 included 5,599 in the CUNY Start intervention group and 5,599 non-CUNY Start students in the comparison group. The authors stated on page 9 that "The overall match included 97 percent of students in the final sample."

Assessments/Attrition: Assessments occurred one year and two years after entry into the university, either as a CUNY Start participant for the intervention group or as a matriculated first-time freshman for the comparison group. The study did not report on attrition but likely excluded dropouts before doing the matching.

Sample:

According to Figure 3, the sample was 10% Asian American, 35% Black, 46% Hispanic, and 9% white.

Measures:

The study examined three primary outcomes based on fulfilling remediation requirements: (1) passed the English gateway course, (2) passed the math gateway course, and (3) passed both gateway courses. The measures came from student records maintained by CUNY.

The outcomes were measured from the point of enrollment in CUNY Start for the intervention group or the freshman year for the comparison group. Since CUNY Start students did not enroll in regular coursework until after the one-semester program, the intervention group had one fewer semester as regular students to take and pass courses than their peers in the comparison group.

Analysis:

The analysis used the matched students with logistic regression models and linear probability models that included fixed effects for college and cohort and controls for demographic characteristics and academic background. It was not possible to control for the outcomes of passing gateway courses, but the academic background controls served as proxies.

Missing Data Method: The study did not appear to have any missing data, but any missing data would have been excluded from the propensity score matching.

Intent-to-Treat: The analysis used all matched participants.

Outcomes

Implementation Fidelity:

Not examined.

Baseline Equivalence:

Not presented

Differential Attrition:

No missing data after matching.

Posttest:

Table A1 shows significant program effects for the matched sample on the three outcomes after one year (or one semester after the program end). The CUNY Start students passed significantly more English, math, and both English and math gateway courses than the matched comparison group. Odds ratios ranged from 1.435 to 2.259.

Moderation tests found similar effects after one year across racial and ethnic group but stronger effects among students with more remedial needs at baseline.

Long-Term:

Table A1 also shows significant program effects for the matched sample on the three outcomes after two years (or three semesters after the program end). The CUNY Start students passed significantly more English, math, and both English and math gateway courses than the matched comparison group. Odds ratios ranged from 1.435 to 1.743.

Study 3

Summary

Polon & Haralampoudis (2021) conducted a quasi-experimental study involving 18,030 students at CUNY who enrolled with remedial coursework needs and self-selected into the intervention or regular coursework. The study used propensity score matching to equalize the conditions on baseline measures and assessed enrollment retention, credit accrual, and degree attainment over the next three to four years.

Polon & Haralampoudis (2021) found that the intervention group, relative to the comparison group, had significantly higher

• Enrollment retention through semester six
• Degree attainment at year four

Evaluation Methodology

Design:

Recruitment: The study sample began with all students who entered CUNY with remedial coursework needs from the Fall 2014 through Spring 2018 cohorts. Eight cohorts of students started in each of eight semesters over the study period. However, the sample was then limited to those students who were retained from semester one to semester two. The authors did not report the total number of eligible students, but the final sample selected for study was 18,030.

Assignment: The intervention group consisted of students who self-selected into CUNY Start, completed the one-semester program, and did not begin remedial coursework until the second semester. The comparison group consisted of CUNY students who did not participate in CUNY Start, began remedial coursework in the first semester, and continued on to the second semester. Both groups thus completed the first semester and enrolled in the second semester, but the groups differed in that the intervention group completed the program while the comparison group completed remedial coursework.

To help make the groups equivalent, the study used one-to-one propensity score matching without replacement. The student characteristics used to calculate the propensity score were age, race, sex, initial semester of enrollment at CUNY, CUNY institution of enrollment, new student code, College Admissions Average, TAP and Pell receipt, high school type, and remedial need in math, reading and/or writing prior to matriculation at CUNY or enrollment in CUNY Start. The matching produced 9,015 students in each condition.

The authors noted considerable overlap of the program with another program ongoing at CUNY, the Accelerated Study in Associate Program (ASAP). About 46% of the CUNY Start participants enrolled in ASAP compared to just 12% of the comparison group students

Assessments/Attrition: Assessments occurred in each of six semesters after the start of the program. The end of semesters three to six came one year or more after the program ended. The use of university records meant that there was no attrition for the eligible sample used in the matching.

Sample:

The matched sample consisted of 58% women, 33% African Americans, 45% Hispanics, and 11% whites. The sample averaged 23 years of age.

Measures:

The study examined three outcomes: 1) the rate of semester-to-semester retention, 2) the number of credits attempted and earned, and 3) the rate of degree attainment. The measures came from university records and appear to be cumulative (i.e., credits and degrees earned in early semesters are counted for later semesters).

Analysis:

The main analyses compared proportions across groups and included significance tests. One mention was made of multivariable regression analyses that controlled for the characteristics used to construct the propensity scores. The outcomes did not exist at baseline, but the model controlled for several baseline measures of past academic performance.

Missing Data Methods. The matched sample had no missing data.

Intent-to-Treat: All matched participants were included in the analysis, but the intervention group included only those students who finished the program and enrolled for the following semester. The comparison group was similarly limited to those who finished the first semester coursework and enrolled for the following semester.

Outcomes

Implementation Fidelity:

Not examined.

Baseline Equivalence:

The balance table in the Appendix reports the standardized mean difference between the matched conditions for the baseline measures. The differences were small, with only one of 34 being greater than .10. Significance tests for differences were not reported.

Differential Attrition:

No attrition

Posttest:

The study could not test for condition differences in persistence at posttest, as only those persisting from semester 1 to semester 2 were included in the sample. Persistence at posttest was therefore 100% for both conditions. For the other two outcomes, the intervention group had fewer credits and degrees than the comparison group.

Long-Term:

Table 2 shows that the persistence rate was significantly higher for the intervention group than the comparison group in semesters 3, 4, 5, and 6. The multivariable estimates indicated that CUNY Start students were 4.8 percentage points more likely to reenroll in their third consecutive semester at CUNY than the matched comparison students.

For the other two outcomes, CUNY Start participants had one less semester to enroll in credit-bearing courses and complete degrees than the comparison group during the span of the study. For both outcomes, CUNY Start students began at lower levels but increased at faster rates and closed the credit accrual gap and degree attainment gaps. For degree attainment in year four, the intervention group had a significantly higher percentage of students with degrees.