Year Up

Study 1

Summary

Fein and Hamadyk (2018), Fein et al. (2021), and (Fein & Dastrop, 2022) recruited a sample of 2,544 young adults located near eight program offices. Local staff randomly assigned participants at a 2:1 ratio to the intervention (N = 1,669) and control groups (N = 875). Survey data of participants were collected at 18 months, three years, and six years after randomization. Administrative records were also collected that followed participants for up to three years, five years, and seven years after randomization. The primary measures focused on earnings. Secondary and exploratory measures included employment, college enrollment, financial hardship, marriage and childbearing, and personality traits.

Fein and Hamadyk (2018), Fein et al. (2021), and (Fein & Dastrop, 2022) found that, compared to the control group, the intervention group showed significantly more

• Employment (3 and 5 years after program uptake)
• Average quarterly earnings (3, 5, and 7 years after program uptake)
• College enrollment (3 years after program uptake)

Risk and Protective Factors

• Increased health insurance rates (3 years after program uptake)
• Reduced financial hardship (3, 5, and 6 years after program uptake)
• Decreased public benefit (5 years after program uptake)
• Improved living arrangements (5 years after program uptake)
• Reported life changes (5 years after program uptake)
• Improved career pathways employment and self-assessed career development (5 years after program uptake)

Design:

Recruitment: Year Up's eight core offices recruited, screened, and selected 2,544 young adults ages 18-24 for the evaluation. To allow for experimental assignment, local admissions teams recruited three eligible applicants for every two open program seats. Each office enrolled at least two cohorts from January 2013 to August 2014. Study locations included Atlanta, the San Francisco Bay area, Boston, Chicago, Washington DC, New York City, Providence, and the Puget Sound area. The young adults eligible for the program had a high school diploma or equivalent and were judged by local staff as likely to benefit from the program. Many had recently experienced financial hardship and very few were employed or in school.

Assignment: Local program staff used an online lottery tool developed by the research team to randomly assign participants within each local site at a 2:1 ratio to the intervention and control groups. Intervention group members (N = 1,669) had the opportunity to participate in Year Up but were not required to enroll. Control group members (N = 875) could not participate in Year Up for three years but received a list of other services available in the community. About 3.4% of the control group joined the program during or after the embargo period but had only a small potential influence on the results, and the analyses kept them in the control group.

Assessments/Attrition: For Fein & Hamadyk (2018), participants completed a baseline survey and an 18-month follow-up survey (about 6 months after the end of the year-long program). The response rate for the follow-up survey was 78% for the treatment group and 73% for the control group. Item non-response among survey responders was under 4%, except for parental college attendance (6.0%), typical high school grades (7.2%), family income (9.5%), and expected near-term future work hours (6.0%).

Additional data from administrative records extended the follow-up period to more than one year after the program end. Data on new hires (from the National Directory of New Hires) accessed in April 2018 covered a period extending from "two quarters before random assignment to three years (12 quarters) after random assignment" (p. 35). Data on college enrollment accessed in April 2017 covered "nearly three years (11 quarters) after random assignment" (p. 35). Data from the administrative records were complete for 98% of the sample (Fein & Hamadyk, 2018).

For Fein et al. (2021), participants completed a baseline survey and a three-year follow-up survey (about 2 years after the end of the year-long program). The response rate for the three-year follow-up survey was 71% overall and the median response occurred at 39 months after random assignment in both the treatment and control groups. Similar to the earlier report, nonresponse rates were less than 5 percent except for parental college attendance (10%), race (9%), Hispanic origin (8%), family income (9%), receipt of WIC or SNAP (7%), and receipt of other public assistance (10%).

Data from a January 2020 extract of earnings data cover a period extending from two quarters before random assignment to five years (20 quarters) after random assignment for all sample members. Since administrative earnings records extend to mid-2019, they do not cover the economic downturn resulting from the coronavirus pandemic. Meanwhile, postsecondary administrative data from a July 2019 extract cover college enrollment outcomes for a five-year (20-quarter) follow-up period following random assignment. Consistent with the earlier report, data from the administrative records were complete for 98% of the sample (Jenkins et al., 2021, Appendix D.

Six- and Seven-Year Follow-Up (Fein & Dastrup, 2022): Assessments came from a six-year survey (median of 70 months after assignment) and from seven years of records. The response rate was 66% for the survey and 100% for the records.

Sample: All sample members were aged 18-24. A majority of sample members identified as non-Hispanic Black (54%) or Hispanic of any race (31%), while 5.5% identified as White, non-Hispanic and 8.8% identified as "any race, non-Hispanic". Men (59%) outnumbered women (41%), though women were well-represented for a tech-focused training program. Most sample members (68%) lived with their parents, and few (9%) had children. Many struggled in high school: 40% reported usual grades of C or below, and only 10% reported usually receiving As. About half had attended some college. Nearly two-thirds (63%) were in families with annual incomes below $30,000.

Measures: The outcome measures, more than 45 in total, came from two independent sources, survey self-reports, and administrative records. Data on new hires came from extracting records from the National Directory of New Hires (NDNH). The database covered most employment but missed self-employment and independent contracting. Data on college enrollment came from the National Student Clearinghouse (NSC). The database included nearly all private and public 4-year schools and public 2-year schools. Coverage was less complete for private 2-year schools and for-profit 2-year and 4-year schools. The study provided little information on the validity of the survey measures, but most appeared straightforward.

Earnings and Employment: Most measures of earnings came from administrative records. The single measure identified by the researchers as primary was the average quarterly earnings in the 6th and 7th quarters after random assignment (Fein & Hamadyk, 2018) and five years (20 quarters) after random assignment for Fein et al. (2021). Four secondary measures included the annual earnings in years 1, 2, 3, and 5 and employment (Fein & Hamadyk, 2018). A total of 17 measures came from the survey. There were three indicators of career pathways employment (working for $15/hour or more, in a job requiring mid-level skills, and in a program-targeted occupation) and three measures of self-assessed career progress (perceived career progress, confidence in career knowledge, and access to career networks). The surveys also provided 11 measures of earnings, hours worked, and hourly wages.

Post-Secondary Education: The seven measures from administrative records covered quarterly college enrollment, quarterly cumulative college enrollment, and yearly college enrollment. The five measures from the survey included credits received and credentials awarded from college, other institutions, licensing, and any source. The measures in Fein & Hamadyk (2018) were described as exploratory because they were not expected to show effects until later assessments (see the six- and seven-year measures below).

Other Outcomes: The 11 measures from surveys reported in Fein & Hamadyk (2018) included dependence on public assistance, financial hardship, access to health insurance, marriage and childbearing, grit, savvy, and core self-evaluation. These measures were described as exploratory because they were not expected to show effects until later, may have effects in unknown directions, or were of unknown validity. In addition to details on employment and related financial circumstances, the survey reported in Fein et al. (2021) assessed educational history, psycho-social skills, and other outcomes.

Six- and Seven-Year Follow-Up (Fein & Dastrup, 2022). This report examined more than 80 outcome measures but also sought to minimize false positives by distinguishing between confirmatory, secondary, and exploratory outcomes. The single prespecified confirmatory outcome was average quarterly earnings in the 23rd and 24th follow-up quarters. The secondary outcomes included other measures of quarterly earnings, employment, and occupation. The exploratory outcomes included measures of employment and earnings in a variety of years and quarters; college enrollment and credential receipt; various aspects of financial well-being; and health, stress, living arrangements, and childbearing.

The measures came from a survey completed roughly six years after random assignment and from administrative records drawn from the National Directory of New Hires and the National Student Clearinghouse for a seven-year period. As in previous reports, this report provided little information on the validity of the survey or administrative measures, but most appeared straightforward.

Analysis: Given the 28 baseline covariates, the analysis reported in Fein & Hamadyk (2018) used a multi-step control strategy. It first regressed each outcome on a set of baseline variables for the control group. It next used the regression results to calculate predicted outcome values for the two conditions. The final steps calculated the average difference between actual and predicted values for each condition and, to assess the program impact, subtracted the average condition differences from one another. It appears from Appendix A.2 of Fein & Hamadyk (2018) that the covariates included baseline outcomes.

A technical report published by Judkins et al. (2021) describes the approach to covariate control for the Fein et al. (2021) study, which included some procedural changes from the Fein & Hamadyk (2018) report. The authors selected covariates using the "least absolute shrinkage and selection operator" (LASSO) technique with "10-fold cross-validation" (see page 9 of Judkins et al., 2021 for more details). A set of covariates across three domains were selected. The three domains included: (1) analyses of employment and earnings outcomes that are conducted on the dataset of merged data from the three-year follow-up survey and the National Directory of New Hires (NDNH); (2) analyses of education outcomes (whether based on the survey, National Student Clearinghouse, or local or state college records); and (3) analyses of all other outcomes (most of which concern personal and family well-being and economic independence). The pool of potential covariates was the same for all three domains with one important exception: indicators of pre-baseline earnings based on NDNH data were only allowed in analyses of NDNH-based outcomes. Exhibit A-3 (p. 12-13; Judkins et al., 2021) shows the covariates selected with the LASSO procedure, which appeared to include baseline outcomes.

For both reports (Fein & Hamadyk, 2018; Fein et al., 2021), tests relied on different significance levels, depending on the outcomes. The analyses of the researcher-identified primary and secondary outcomes, called the confirmatory and secondary analyses, used one-tailed tests but were limited in number to minimize the problem of multiple comparisons. The more extensive exploratory analyses used two-tailed tests. But the tables present sufficient information to apply two-tailed tests at .05 to all results.

Six- and Seven-Year Follow-Up (Fein & Dastrup, 2022). The authors reported regression-adjusted means that controlled for baseline covariates (a subset selected by optimization procedures from a list of 20+ measures contained in Appendix A of Judkins et al., 2022). The tests for differences between the means were one-sided for the confirmatory and secondary outcomes and two-sided for the exploratory outcomes, but the reported probabilities allow for application of two-sided tests to all results. The authors also noted that the reported results did not adjust standard errors to account for variation in effects across offices. Checks estimating standard errors with a cross-office variance term for key outcomes found that all earnings impacts remained highly significant.

Intent-to-Treat: The study included participants assigned to the intervention group but who did not enroll in the program. For those lost to follow-up, the analyses applied weights to adjust for differential nonresponse. In addition, missing data for item non-response was imputed using a weighted "hot deck" matching procedure (Appendix A.1 of Fein & Hamadyk, 2018; Appendix A.1 of Judkins et al., 2021).

Fein & Dastrop (2022) kept participants in their originally assigned condition, regardless of actual participation or non-participation in the program. For the survey data, the analysis used two procedures described in Appendix B of Judkins et al. (2022) to deal with missing data. The first replaced missing items for otherwise complete data; the specifics depended on the outcome measure and the extent of missing data. The second applied weights to adjust for differential attrition or non-response; it did not impute missing data but gave additional weight to the participants most similar to dropouts.

Outcomes

Implementation Fidelity: Nearly all treatment group members (96%) accepted the offer to enroll and began the program. About 81% of the sample and 85% of enrollees completed the six-month learning and development phase. Nearly all those completing the learning and development phase received internships, and 75% of the treatment group members finished the program.

A detailed implementation analysis found that Year Up generally implemented its services with high fidelity to the program model. Overall, treatment group members received substantially higher levels of training and support than control group members. However, the eight offices varied in completion of the program components (range = 61% to 82%).

Baseline Equivalence: Using all 2,544 randomized participants, tests for condition means on 28 baseline measures of sociodemographic characteristics, outcomes, and covariates produced no significant differences (p < .05).

Differential Attrition: The conditions had roughly similar survey response rates of 78% for the treatment group and 73% for the control group (Fein & Hamadyk, 2018). Exhibit D-3 compared the condition means for the baseline measures using all participants and survey respondents. There were no significant differences at the .05 level for the 28 baseline measures in either the full or the analysis sample. The analyses still used a weighting procedure to adjust for nonresponse (see Appendix D.3 of Fein & Hamadyk, 2018). Further, checks in Exhibit D-4 (Fein & Hamadyk, 2018) demonstrated that, for outcome measures available for the full sample, analysis of the subsample of survey respondents produced results similar to those for the full sample. Of note, the primary outcomes relating to earnings were assessed via archival data which had a much higher retention rate.

For the 3-year follow-up (Fein et al., 2021), the conditions had slightly different response rates of 74% in the treatment group and 67% in the control group. Similarly, Fein et al. (2021) explain that: "Analyses of survey data incorporated weights to adjust for differential nonresponse across groups of study participants" (p. 25). Details are listed in Appendix B.2 of Judkins et al., (2021). And consistent with the earlier report, the primary outcome (i.e., earnings) was evaluated using administrative data with a high retention rate.

For the six-year survey, Fein & Dastrop (2022) reported only that the treatment and control groups had 68% and 62% retention, respectively. The analysis of non-response in Judkins et al. (2022) used complete administrative data to compare condition differences in the outcomes for the full sample to differences for the subsample of survey respondents. It found substantial evidence of bias. But, as described in Appendix B.2 of Judkins et al. (2021), the use of weighting helped adjust for differential non-response.

Posttest: Three years after baseline, Fein & Hamadyk (2018) found that approximately 30 of the 45 outcomes showed significant effects, including the primary and secondary measures of earnings and employment.

Earnings and Employment: The primary outcome - average quarterly earnings in the 6th and 7th quarters after random assignment - was significantly higher for the intervention group than the control group (p < .05, two-tailed test). More detail in Exhibit 6-2 of Fein & Hamadyk (2018) revealed significantly lower earnings in year 1, when the program was ongoing and intervention participants were receiving stipends, but significantly higher earnings in years 2 and 3. The same pattern emerged for employment, with significant positive effects in later years. Tests for the survey measures showed significant intervention effects in 16 of 17 outcomes. The authors noted (Fein & Hamadyk, 2018, p. 7) that the increase in earnings came from more hours worked and higher wages rather than from higher rates of employment.

Post-Secondary Education: Tests using the seven administrative measures showed generally positive effects three years after baseline. The intervention group had significantly higher college enrollment in year 1, when the program was ongoing, and significantly lower enrollment in year 2. However, when measuring cumulative months of full-time enrollment (Exhibit 6-7 of Fein & Hamadyk, 2018), the intervention group had significantly higher scores than the control group in all years. Tests for the five survey measures found three significant effects on credits received, a licensing credential received, and a credential received from any source (18 months after baseline).

Other Outcomes: Tests for 11 self-reported outcomes found five significant effects of the intervention 18 months after baseline: having health insurance, receiving less cash or in-kind support, experiencing less financial hardship, perceived savvy, and core self-evaluation (Fein & Hamadyk, 2018).

Long-Term: The effects of the program on earnings and cumulative college enrollment were maintained in years 2, 3, 5, 6, and 7 -- well after the 1-year follow-up period.

Earnings and Employment: Fein et al. (2021) found the large positive earnings impacts persisted undiminished to the end of the five-year follow-up period. That is, the impact on average quarterly earnings in follow-up Quarters 12 and 13 (the sole confirmatory outcome for Fein et al., 2021) was large, at $1,857, and statistically significant. Average earnings were $6,782 for treatment group members and $4,925 for control group members. Impacts of about $2,000 per quarter extended to the end of the five-year follow-up period (Exhibit ES-1 in Fein et al., 2021). The result indicates that impacts persisted virtually undiminished after reaching $1,895 in Quarters 6 and 7-the confirmatory outcome in the short-term report (Fein and Hamadyk 2018).

Secondary analyses estimated impacts on annual earnings for up to five years after random assignment. Fein et al. (2021) report that earnings impacts turn positive and large in Year 2 ($5,267). The size of the impact appears to grow with each additional year of follow-up (reaching $7,830 in Year 5), but statistical tests do not find the increases to be statistically significant (Exhibit 3-2).

Consistent with the earlier report (Fein & Hamadyk, 2018), Fein et al (2021) report survey results that suggest the increase in earnings came from more hours worked and higher wages rather than from higher rates of employment. In contrast to the survey, however, administrative records show small positive employment impacts in a number of quarters during follow-up years 4 and 5 (Exhibit 3-4, Fein et al., 2021). The impacts are small (only 3 percentage points) but statistically significant.

Meanwhile, survey results show impacts in favor of treatment on 5 of 6 indicators of career pathways employment and 2 of 3 indices of self-assessed career development.

Post-Secondary Education: From Year 3 on, college enrollment rates (an exploratory outcome) were virtually identical in the treatment and control groups. At the time of the three-year follow-up survey, Fein et al. (2021) found few members of either group had earned an associate degree or higher, although the fraction was lower for treatment (4 percent) than control (8 percent) group members. Administrative records show a smaller, but statistically significant, two percentage point reduction in degree receipt (see Supplemental Exhibit S-4 of Fein et al., 2021). In addition, whereas in the 18-month survey (Fein & Hamadyk, 2018), treatment group members reported receiving an average of four more credits than control group members did (12 and 8 credits, respectively), the difference had disappeared by the time of the three-year follow-up (see Exhibit 4-2 of Fein et al., 2021): treatment and control group members reported receiving a similar number, averaging 16 and 18 credits, respectively.

The significant reduction in associate degree attainment could be construed as an iatrogenic effect. As an alternative explanation, the authors posit that, because the treatment condition participants were making more money than the control condition, they were less motivated to pursue further education.

Other Outcomes: Results show signs of impacts in secondary "other" outcomes, but effects were not widespread. For example, findings from the three-year survey (Fein et al., 2021) show reductions in public benefit receipt, debt, and financial hardship. Average household income did not increase, which the authors conclude could be because increased earnings both reduced some households' public benefit eligibility and allowed some young adults to live independently. The treatment had a few small effects on living arrangements, but no impacts on two psycho-social outcomes (academic self-confidence and future educational aspirations). In addition, treatment had a small reduction in the average number of reported life challenges but had no impact on perceived stress, social support, core self-evaluation, and grit.

Subgroup findings (exploratory outcomes): Fein et al. (2021) found that 5-year impacts on quarterly earnings were larger for young adults receiving mostly A's or B's in high school ($2,178) than for those receiving mostly C's or below ($1,388), and larger for those who identify as White ($2,931) than for those who identify as Black ($1,570). Variation in impacts across the eight local offices (from $956 to $4,811, with most offices in the $1,348 to $1,807 range) also suggests that local conditions might matter.

Six- and Seven-Year Follow-Up (Fein & Dastrop, 2022). The confirmatory outcome of earnings in quarters 23 and 24 was significantly higher ($1,895) for the treatment group than the control group. Nearly all of the secondary outcomes, which related to earnings for other quarters, occupation, and financial well-being, showed significant program benefits; the treatment group did significantly better than the control group on 16 of 17 tests (two-sided, p < .05). Some of the exploratory outcomes showed significant effects; the treatment group did better than the control group on 22 of 66 tests.

The most consistent results occurred for measures of earnings at seven years, and job quality and financial status at six years. Generally weak results for measures of employment and education suggested that the higher earnings brought about by the program came from better jobs rather than higher employment or education. Null results occurred for exploratory outcomes relating to psycho-social well-being, family formation, or self-assessed health.

Exploratory moderation analyses, based on a large number of tests, found that the program most benefited participants with better high school grades and some college, older ages, and fewer depressive symptoms.

The COVID pandemic began during data gathering, and checks found that program participants enjoyed modest protection from lower earnings and employment during the pandemic. For the confirmatory outcome, the pandemic reduced average quarterly earnings in follow-up quarters 23-24 for all participants but did not change the advantage of the treatment group over the control group.

Study 2

Summary

Roder and Elliott (2011, 2014) recruited 195 youths (ages 18-24) in three Eastern cities in the United States for the study and randomly assigned them to the program group or a no-treatment control group. The youths were followed for three years after the end of the program. The analysis examined self-reported measures of earnings, employment, and college attendance.

Roder and Elliott (2011, 2014) found that, compared to the control group, the intervention group showed significantly more

• Full-time employment and IT and finance jobs
• Earnings and hourly wages

Risk and Protective Factors

• College financial assistance and interest in attending college

Evaluation Methodology

Design:

Recruitment: Program staff in three cities, Boston, New York City, and Providence, Rhode Island, invited eligible young people to take part in the program. The recruitment in 2007 took place shortly before the recession of 2008. Eligible youths came from low-income backgrounds, were ages 18-24, had a high school degree or GED, and showed strong motivation to improve their opportunities with an intensive, full-year training. A total of 195 agreed to join the study.

Assignment: The research team randomly assigned the participants based on the capacity of the programs in the three cities. Of the 195 youths in the sample, 135 were randomly selected for the intervention group and 60 were randomly selected for the control group. Control participants were told that they were being placed on a waiting list, could re-apply to the program after ten months, and were allowed to pursue employment or postsecondary education or training elsewhere. However, 29% of the control group members eventually participated in the program, which affected the long-term comparisons.

Assessments/Attrition: Roder and Elliott (2011) mentioned that the last survey occurred between 24 and 30 months after random assignment (or 12 and 18 months after the program end). Figures 2 and 3, however, reported quarterly data over two years, suggesting either ongoing surveys after randomization or retrospective reports. The follow-up survey included 164 youths, with completion rates of 84% overall, 89% for the intervention group, and 73% for the control group.

Roder and Elliott (2014) followed the participants for an additional two years, three years after the end of the program. The response rates of the follow-up survey were 73% overall, 76% for the intervention group, and 68% for the control group.

Sample:

Of the 164 young people in the analysis sample, 50% were African American and 34% were Latino. More than half (57%) were male, 81% lived with a parent or guardian, and 18% lived in public housing. All had at least a GED or high school degree, but 8% had a criminal conviction and English was a second language for 15%.

Measures:

For the one-year follow-up (Roder and Elliott, 2011), program staff collected data on students who graduated from the program, while a survey firm conducted interviews with members of the control group and with members of the intervention group who dropped out of the program. For the long-term follow-up (Roder and Elliott, 2014), a single survey firm collected data on all available participants. The study provided no details on the reliability or validity of the self-reports from participants, but the measures were straightforward. The outcomes included:

• Total earnings
• Employment, including full-time employment and number of jobs
• Hourly wages
• Job type (e.g., IT and finance)
• Employer-provided benefits
• College attendance

Analysis:

The analyses simply compared condition means and percentages, without any controls.

Intent-to-Treat: The analyses included all participants with follow-up data, regardless of whether or not they ever attended or graduated from the program. For the long-term data, after many of those initially assigned to the control group had entered and completed the program, treatment-on-the-treated results were reported along with ITT results.

Outcomes

Implementation Fidelity:

The study did not measure the quality of implementation but reported that, of the 120 intervention group members, 90% attended part of the program and 64% graduated on time in July 2008.

Baseline Equivalence:

The study did not examine baseline equivalence of the conditions.

Differential Attrition:

For the one-year follow-up (Roder and Elliott, 2011), the conditions differed substantially in attrition rates - 11% for the intervention group and 27% for the control group. Footnote 2 examined the significance of differences in the attrition rates between the intervention and control group members by baseline characteristics. There were significant differences on three factors. Attrition was higher among control group members than among intervention group members for those ages 18-19, African Americans, and those with English as the primary language.

For the long-term follow-up (Roder and Elliott, 2014), the conditions differed moderately in attrition rates - 24% for the intervention group and 32% for the control group. The appendix examined the significance of differences in the attrition rates between the intervention and control group members by baseline characteristics. There were significant differences on two factors. Attrition was higher among control group members than among intervention group members for those ages 18 to 19 and those with a GED rather than a high school diploma.

Posttest:

The results in Roder and Elliott (2011) covered the quarterly periods during the one-year program and the one-year post-program period. During the program period, when the intervention group was going through the training, the control group had higher employment and earnings. However, during the second year after random assignment, relative to the control group, the intervention group showed significantly more:

• fulltime employment (but not overall employment),
• annual earnings in the second and third quarter,
• earnings per hour, and
• more jobs in information technology and investment operations.

Long-Term:

The results in Roder and Elliott (2014) covered all four years of the study but focused on the three years after the program end. The comparisons were limited by the entrance of 29% of control participants into the program, which likely lowered employment and earnings of the control group during the training and raised employment and earnings afterward. As a point of comparison, intervention participants made approximately $2000 less per quarter, compared to control participants, while completing the intervention.

Figure A1 in the appendix lists the ITT effects for 25 outcomes, with only three reaching significance at the .05 level, while the text listed some additional significant effects. Relative to the control group, the intervention group showed significantly more:

• holders of a single job (but not higher employment rates or hours worked),
• earnings in years 2-3 after the program but not year 4,
• jobs in information technology and investment operations,
• recipients of educational financial assistance, such as loans, grants, or scholarships, and
• interest in attending college (among those not attending college).