Valley Initiative for Development and Advancement (VIDA)

Study 1

Summary

Rolston et al. (2017, 2021) and Rolston and Walton (2022) randomly assigned 958 individuals to either the treatment or the control group. Data were collected via self-report surveys and administrative records measuring postsecondary education and individual and family well-being outcomes at 24 months (Rolston et al., 2017), 3 years (Rolston et al., 2021), 3-5 years (Rolston et al., 2021), and 6 years (Rolston & Walton, 2022) after random assignment.

Rolston et al. (2017, 2021) and Rolston and Walton (2022) found that, compared to the control group, treatment group members earned significantly more

• College credits and credentials over an initial 24-month follow-up period
• College credentials requiring a year or more of training, a confirmatory outcome, three years and six years after random assignment
• Associate degrees (or higher) three years and six years after random assignment.

Evaluation Methodology

Design:

Recruitment: Participants in the study had to be unemployed, underemployed, meet federal poverty income levels, or be on public assistance. Additionally, they had to be residents of the Rio Grande Valley, 18 years or older, and eligible to work in the United States. The program required at least a high school diploma or GED and enrolled participants with prior college experience. Individuals had to be committed and able to enroll full time in college in an occupational program. To meet its recruitment goals, during the period of the study, the intervention was extensively marketed through media, partners and word-of-mouth referrals.

Assignment: Between November 2011 and September 2014, the 958 program applicants were randomly assigned to either the treatment (n=478) or the control group (n=480). The control group received normal services and were embargoed for two years from treatment. Control group members had access to other programs and services available in the community, including training at the five colleges with which the program partners, but after two years, control students could apply again for the treatment and enroll if they met eligibility criteria.

Assessments/Attrition: Assessments came from a 20-month survey, a 3-year survey, and a 6-year survey, and from administrative records at 1 year, 3 years, 4 years, 5 years (for the subset of early cohorts who started five years previous), and 6 years. The 20-month survey had an 85% completion rate, the 3-year survey had a 76% retention rate, and the records data had a 100% completion rate (excepting the exclusion of late cohorts at the 5-year follow-up). The 6-year survey had a 77% response rate, and the records data again had a 100% completion rate.

Sample: Approximately half the sample had annual household incomes of less than $15,000, and more than 85 percent had incomes less than $30,000. About two-thirds received benefits from the Supplemental Nutrition Assistance Program (SNAP, formerly known as Food Stamps) or Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). About two-thirds also reported experiencing financial hardship in the past year. More than 60 percent were age 25 and older, and more than one-fifth were age 35 or older. Virtually all had at least a high school degree or equivalent, and more than half had a year or more of college. Most study participants (about 70 percent) were female. The vast majority (95.8%) were Hispanic, and most (64.9%) were not working at the time of random assignment.

Measures:

The study identified confirmatory, secondary, and exploratory outcomes, which came from surveys and administrative records, both independent sources. The measures have good face validity, but the records data has some gaps. Educational records come from four local colleges and universities and capture 96% of enrollments; earnings records come from state unemployment systems and miss employment outside the formal sector.

The three confirmatory outcomes included measures of:

• total academic and technical credits earned within 2 years
• receipt of a college credential requiring a year or more of study
• average quarterly earnings within three years

The 14 secondary outcomes included measures of college degrees, credits, certification, and enrollment; employment, earnings, and job skills; career knowledge and supports; and family financial well-being.

The 39 exploratory outcomes included measures of confirmatory and secondary outcomes over a four- and five-year period plus measures at the 3-year survey of job quality, family financial status, parent engagement, child outcomes, childbearing, and living arrangements.

The 6-year follow-up (Rolston & Walton, 2022) listed two confirmatory outcomes:

• receipt of a college credential after eight or more months of full-time-equivalent (FTE) college enrollment by the 24th follow-up quarter (education domain),
• average quarterly earnings in the 23rd and 24th follow-up quarters (employment domain).

A count from the tables listing results indicates tests for approximately 16 secondary outcomes relating to college degrees, enrollment, quarterly earnings, and full-time work. The tables also list about 77 exploratory outcomes. The exploratory outcomes at six years of follow-up included measures of employment and earnings in a variety of other years and quarters, additional measures of college enrollment and credential receipt, measures of financial well-being, and several measures related to parenting and child well-being.

The data for the 6-year follow-up came from survey results and from two sources of administrative records (both matched to the student sample in June 2021). The first source, the National Student Clearinghouse (NSC), had data on student enrollment, degrees earned, and other credential completion from most U.S. institutions of higher education. The second source, the National Directory of New Hires (NDNH), aggregates wage records reported on a quarterly basis to states by employers.

Analysis: In conducting impact regression analyses, the research team controlled for baseline differences by using baseline values as covariates. For confirmatory outcomes, authors applied a one-tailed test of significance because each has a hypothesized direction (an increase in the average level of each outcome).

Intent-to-Treat: The authors reported they followed an intent-to-treat protocol. The 20-month survey data (Rolston et al., 2017) were used to help impute values for missing data on job and education spells from the three-year follow-up survey (Rolston et al., 2021). In addition, Rolston et al. (2021) used National Student Clearinghouse data to adjust for non-response bias in the survey (see Appendix C, Rolston et al., 2021).

Outcomes

Implementation Fidelity: No quantitative information about fidelity was provided. However, implementation data showed most treatment group members received a substantial dose of individual and group counseling. Almost all participants (99 percent) received assistance with tuition and expenses related to attending and completing school, and the program provided an average of almost $7,000 in direct financial assistance per participant within 24 months of random assignment. Almost all treatment group members (98 percent) participated in an education or training activity, which included both basic skills education in the College Prep Academy and occupational training. Ninety-seven percent of all treatment group members participated in occupational training, including those who advanced from the College Prep Academy into college-level programs of study. Ninety-one percent of treatment group members earned at least one college credit, and 66 percent completed 30 or more college credits (by some standards the equivalent of a year of full-time college).

Baseline Equivalence: Significant differences were observed in 2 of 11 baseline characteristics measured: treatment group members were more likely than control group members to have college experience and to be working 20 hours or more (see Exhibit 2-2, p. 17 of Rolston et al., 2017 or Exhibit 1-1, p. 6 of Rolston et al., 2021). There were no baseline differences in age, gender, race/ethnicity, income, receipt of public assistance, and expected work hours in the next few months.

Differential Attrition: There was no attrition for primary and secondary confirmatory outcomes (administrative records were used). Differential attrition tests for survey data (measuring exploratory outcomes) were not reported. Rolston & Walton (2022) noted only that retention rates were 81% for the treatment group and 73% for the control group.

Posttest: The average length of participation in VIDA is 18 months and the Rolston et al. (2017) evaluation was conducted around 24 months post randomization. However, because some treatment participants first entered the College Prep Academy, others were newly enrolling in college, and still others were already enrolled in an occupational program, the program expected that only a subset of treatment and control group members would have had time to complete their programs during the 24-month follow-up period. Indeed, two years after random assignment, 42 percent of the treatment group was still attending training. Among those still attending, about half were still working towards a credential while the other half had obtained a credential, suggesting they continued in their educational pathway to attain further certificates or degrees.

Thus, Rolston et al. (2017) reported on college credits earned as the relevant confirmatory posttest outcome and attainment of a college credential as a secondary outcome. As reported in Exhibit ES-1 (p. viii), treatment significantly increased the total college credits earned within 24 months of random assignment by roughly six percentage points: 33.1 credits for the treatment group compared to 27.5 credits for the control group (p < .001 reported for a one-tailed test). Additionally, 53 percent of treatment group members earned a college credential compared to 45 percent of control group members (an 8 percentage point difference; p = .003, one-tailed test).

Long-Term: To summarize the long-term results, Rolston et al. (2021) found that the intervention, relative to the control group, significantly improved 1 of 2 confirmatory outcomes (college credentials but not earnings) and 4 of 13 secondary outcomes (associate degree, any college credit, enrollment, and number of college credits). The 39 exploratory outcomes were consistently and significantly improved by the intervention in the education domain but not improved in the earnings domain and rarely improved in the life outcomes domain.

Specifically, three years after random assignment, Rolston et al. (2021) found that 61 percent of the treatment group earned a college credential requiring a year or more of training compared to 52 percent of the control group (a statistically significant difference of 9 percentage points; Exhibit ES-1; p = .001 reported for a one-tailed test). In addition, compared to control, three years after random assignment, treatment significantly increased receipt of an associate degree or higher by 7 percentage points (p = .007 reported for a one-tailed test) and "any college credential" by 13 percentage points (p = .001 reported for a one-tailed test). These results are reported in Exhibit 3-1 (p. 23). However, there were no long-term impacts on earnings, which was a confirmatory outcome (both treatment and control groups earned somewhat over $6,000 in quarters 12-13 after random assignment; see Exhibit 4-1, p. 30).

At three years after random assignment, Rolston et al. (2021) found no statistically significant impact on average hourly wages or employment, including working in jobs requiring at least mid-level skills. The evaluation also conducted exploratory analyses of indicators of job quality, such as working at least 32 hours per week, working in a job that offers health insurance, and working in a job with a supportive environment. No impacts on these job quality indicators were detected.

At three years post randomization, there were no significant long-term effects on financial outcomes, including reducing student debt or other forms of debt, indicators of financial distress, or receipt of means-tested public benefits. And there was no effect on having health insurance (Rolston et al., 2021).

Sub-group analyses found that three years after randomization, significantly fewer women in the treatment group compared to the control group had a child since study enrollment or were pregnant at the time of the survey, and women in the treatment group were less likely to be living with a spouse or partner compared to women in the control group.

Some analyses reported in Rolston et al. (2021) extend outcomes to 5 years of education records for an early cohort of participants representing about 80% of the total sample. Five years post randomization, findings show treatment students - compared to control - received significantly more 1) college credentials requiring 1 one or more years to earn, 2) associate degrees (or higher), and 3) "any college credential" (see Exhibit 3-1, p. 23).

Six-Year Follow-up (Rolston & Walton, 2022)

The authors first noted that for about 20% of the sample, the follow-up period overlapped with the COVID-19 pandemic. However, because the program had no detectable impact on quarterly earnings in 2019 or 2020, the research team did not further investigate the pandemic's effects on program impacts.

Overall, the program showed strong educational benefits but little impact on employment, earnings, or several measures of well-being. The authors noted that the inability of educational advances to translate into economic improvement was unexpected and puzzling.

In the education domain, the confirmatory outcome of receiving a college credential proved significantly higher in the intervention group than the control group. Additionally, in response to a Blueprints request, the lead author provided results that indicated this effect was consistent across all age groups - ages 18-24, ages 25-34, and ages 35 and older.  Further, six of the seven secondary and exploratory outcomes, which related to enrollment and completion of early credentials, showed significant intervention effects.

In the earnings domain, the confirmatory outcome of average earnings in quarters 23 and 24 did not differ significantly across conditions. Only one of the 50 secondary and exploratory outcomes (access to career network) was statistically significant.

In the other domains, only three of the 36 outcomes were statistically significant. Compared to the control group, the intervention group experienced reduced receipt of means-tested public benefits, and women in the intervention group had a reduced likelihood of childbearing and were less likely to be living with a child.