YouthBuild

Study 1

Summary

Cohen and Piquero (2010) used a quasi-experimental design in which data were collected from 388 individuals participating in the larger YouthBuild (YB) Offender Project in 30 sites from the fourth quarter of 2004 through the second quarter of 2007. Data were collected and reported on a quarterly basis. YB graduates were compared to YB dropouts.

Cohen and Piquero (2010) found, compared to YB dropouts, the graduates of YB were significantly more likely to:

• Graduate from high school or obtain a GED
• Exhibit lower criminal offending rates (convicted or incarcerated for a new crime or had parole revoked from a previous offense)

Evaluation Methodology

Design:

Recruitment: In 2004, the U.S. Department of Labor (DOL) funded YB USA to identify and grant money to local programs to add program participants under an incarcerated youth reentry program. To be eligible, program participants had to fit into one or more of three categories: a) young people who had been referred by the courts to YB as a diversion program to avoid incarceration, including those on probation, b) young people, having served time in prison or jail,  who were referred by the criminal justice system to YB in a coordinated reentry process, including those on parole, or c) young people who found their own way to YB - having been convicted of a crime and served time in prison or jail previously - who still needed education or job-training opportunities. YB USA subsequently awarded grants to 30 local YB program sites that met DOL-established criteria on performance, community linkages, outreach and recruitment capacity, and leadership. Based on the DOL selection criteria, the 30 sites were representative of well-run YB programs and included both urban and rural areas as well as a wide geographic dispersion.

Assessments/Attrition: Data were collected from 388 individuals on a quarterly basis, from the fourth quarter of 2004 through the second quarter of 2007. On average, data were available for 10.3 quarters (about 31 months) - 10.4 quarters for graduates and 10 quarters for dropouts. YB graduates were compared to YB dropouts.

Sample:

Participants were mostly male (85%), non-white (76%) and averaged 19.62 years old. Only 11% had their high school diploma or GED upon program entry. The average household income was $8,784 and only 9% were working at entry. The vast majority (97%) had a prior arrest, 70% had prior misdemeanors, and 46% had prior felonies; 60% had served time in juvenile detention and 40% had been in adult facilities. Only a third were participating in intensive aftercare at the time of entry to the program, and even though 41% had substance abuse problems only 14% were receiving treatment for these issues. Almost two-thirds were living with their parents (63%), while others were in public housing (10%), living by themselves (10%), with their foster parents (4%), in a halfway home (3%), homeless (3%) or living in a group home (2%).

Measures: Outcomes included high school and GED completion as well as conviction for a crime, incarceration, and parole revocation. However, no information was provided on how the data were collected.

Analysis: Analyses of variance were utilized, first, to analyze differences between program graduates and program dropouts. A second set of analyses incorporated probit regressions that controlled for pre-entry criminal record, educational and work status, living situation, several demographic characteristics, and individual program characteristics.

Intent-to-Treat: Analyses used all available data on the 388 program participants, but compared graduates to dropouts as there was no non-treatment control group.

Outcomes

Implementation Fidelity: While fidelity was not reported in the article, the site selection criteria resulted in a sample of primarily high-functioning YB programs.

Baseline Equivalence: Results in Table 1 showed six significant differences (p < .05) in 26 tests. YB graduates came from higher income households, more graduates lived with parents, and fewer lived in halfway houses as compared to YB dropouts. Dropouts had worse criminal records than graduates did: more dropouts had a prior felony, had served time in juvenile detention, and had served time in an adult correctional facility.

Posttest: Based upon the ANOVAs, program graduates fared better in terms of education than those who dropped out of the program. Among those who entered the program without a degree, YB graduates were significantly more likely to graduate from high school or obtain a GED compared to the YB dropouts (58% vs. 18%).

Overall, compared with dropouts, the YB graduates also had lower criminal offending rates as indicated by three "failures" (i.e., convicted or incarcerated for a new crime or having parole revoked from a previous offense).  Based on the quarters after program entry and separately on the quarters after program departure, significantly fewer graduates were incarcerated or had their parole revoked. Additionally, when combined, fewer graduates experienced any of the three failures. Finally, when offending rates by quarter (instead of by student) were assessed, similar results emerged.

Study 2

Miller et al. (2018) presented final impact results. Two other studies with less essential information are mentioned only briefly: Miller et al. (2016) presented interim impact results and Wiegand et al. (2015) presented implementation results.

Summary

Miller et al. (2018) conducted a randomized controlled trial that examined 75 program sites, with 3,929 youth randomly assigned within sites to intervention and control groups. The study followed participants for 12-48 months and used both survey and administrative data to measure education, employment, recidivism, and youth development outcomes.

Miller et al. (2018) found, compared to the control group, the treatment group showed significantly improved:

• Educational outcomes
• Employment outcomes

Evaluation Methodology

Design:

Recruitment: The study began with 74 YouthBuild sites that had received funding from the Department of Labor. Then 60 were randomly selected for study, and two were dropped due to the infeasibility of randomization. Along with these 58 sites, the study added 24 others with a different funding source. However, seven of these were then dropped for not having a suitable control group or planning to shut down, leaving 17 sites. Combining the 58 and 17 sites, the sample included 75 sites.

From 2011-2013, 3,929 participants joined the study through the sites. Recruitment came from a variety of sources, and eligibility was based on being ages 16-24, having dropped out of high school, and falling into at least one risk category (e.g., current or former foster child, disabled, criminal offender, low income, incarcerated parent). Screening ensured participants met minimal academic standards. Screening also came from requiring participants to complete a 10-day Mental Toughness Orientation. An average of one in four recruits invited to the orientation did not complete it.

Assignment: Each site used randomized assignment but determined the timing of the assignment. Of the 75 participating sites, 72 successfully completed random assignment at least once during the study period of August 2011 to January 2013. Sites could bypass random assignment when short of applicants during an enrollment cycle, and three were unable to randomize. All sites performed the randomization by having staff members enter data into an MDRC random assignment system. Each site aimed for a 60:40 assignment ratio but could deviate from the ratio when recruitment numbers were low. Of the 3,929 participants, 69% (2,700) were randomly assigned to the intervention group and 31% (1,229) to the control group.

Some complications followed from the option of each site to choose the timing of the randomization. The study noted that most sites (81%) opted to conduct random assignment before or during the first few days of the orientation. It appears from the article that the sample of 3,929 comprised the full randomized sample rather than the subset of randomized participants who completed the orientation. Correspondence with the first author confirmed that the full sample was used and that all treatment participants actually enrolled in the program after the orientation. It also appears that, given the timing of the randomization, some control participants would have attended part or all of the program orientation, even though they did not get program services after the orientation.

Assessments/Attrition: The study followed participants through 48 months with administrative data, and three surveys were administered at 12, 30, and 48 months after they entered the study. The 48-month survey came about two years after the end of the 24-month program.

The administrative data had little if any attrition. The survey had missing data from two sources. Due to budget constraints, the survey went to a randomly selected subset of 3,436 participants (88%). Among the surveyed subset, response rates were 78-80% across the waves. A total of 2,721 participants provided responses to the 48-month survey.

Sample:

The sample was 64% male, 46% aged 19-21, 63% black, 15% non-Hispanic white, and 15% Hispanic. Only 10% had a high school degree at the program start.

Measures:

For measures of quarterly earnings and employment, administrative records came from the National Directory of New Hires. The data source included workers covered by the unemployment insurance system but not workers with informal and self-employed jobs.

For measures of postsecondary enrollment, administrative records came from the National Student Clearinghouse, which tracks enrollment and degree receipt nationally. Four-year public institutions had the highest coverage (99%) and for-profit institutions the lowest coverage (estimated at 48%).

The surveys included eight measures of youth development (e.g., civic engagement, good health, trust, self-esteem) and 13 measures of self-reported delinquency and risky behavior (e.g., arrests, convictions, incarceration, substance use). An additional 23 survey measures reported in Appendix C included leadership, happiness, social support, living arrangement, receipt of government benefits, and involvement with a gang. Many measures were single items, but for the scaled measures, the study provided no information on reliability or validity.

Analysis:

The analysis used multiple regression models with controls for baseline sociodemographic characteristics and site fixed effect but not for the baseline outcomes. For the analysis of survey outcomes, weights were added to the model to account for varying selection probabilities by cohort and research group.

For missing baseline covariates, the study imputed the sample mean, and the model included dummy variables indicating imputation. Observations with missing values for an outcome were dropped from the impact analysis for that outcome rather than imputed.

Intent-to-Treat: ITT analyses used all available data, including all randomized subjects when administrative data were complete and all those with posttest data when there was attrition. Intervention students who did not complete the orientation were analyzed in their originally assigned condition. Supplemental treatment-on-the-treated analyses were presented in appendix Table B.2.

Outcomes

Implementation Fidelity:

The average fidelity score was 79 out of 100 (Wiegand et al., 2015, p. 37). Although there was a good deal of variation around the average, most sites appear to have implemented the program well. The control group also received many services, but the program group was more likely to have participated in educational, training, and personal development services.

Baseline Equivalence:

Results in Table B.1 using the full randomized sample of 3,939 showed one significant difference (p < .05) in 39 tests.

Differential Attrition:

With no or very little attrition for administrative data, tests for differential attrition focused on the survey data. The study presented multiple types of analyses.

• A comparison of baseline measures for the full randomized sample with the randomly selected subset of those surveyed (Table B.3) found no significant (p < .05) differences in 42 tests.
• A comparison of the response rates for the conditions showed 80% for the program group and 76% percent for the control group. Although statistically significant, the 4% difference was small.
• A comparison of baseline measures for respondents and non-respondents using the full randomized sample of 3,929 (Table B.4) showed significant differences (p < .05) for gender, race, education, and housing status.
• A comparison of baseline measures for the two conditions using the analysis sample of survey respondents (N = 2,695) in Table B.5 showed one significant difference (p < .05) in 40 tests (92.4% with high school degree or equivalency for the intervention group versus 89.7% for the control group).
• A comparison of results of administrative data for the full sample and the survey analysis sample (Table B.6) showed no significant difference in 16 coefficients for the two samples.

Posttest:

The interim results reported in Miller et al. (2016) came six months after the maximum length of the program and can serve as the posttest. To summarize, the program significantly increased participation in education and training, including earned high school equivalency credentials, enrollment in college, and participation in vocational training. It also led to a small increase in wages and earnings and civic engagement, particularly volunteering, but it had few effects on other measures of youth development or attitudes. The program had few effects on criminal justice involvement.

Long-Term:

The long-term results came at 48 months, about two years after the end of the program.

Education (Table 3.1): In 23 tests, 11 statistically significant effects showed better treatment group outcomes, relative to the control group, for 1) survey-reported outcomes of high school diploma or equivalency, enrollment in vocational schools, and post-secondary enrollment, and 2) administrative outcomes of attended college and received a degree. However, the rates of college enrollment were low, peaking at 10% in the fourth quarter and declining to the level of the control group by the 12th quarter.

Employment and Earnings (Table 3.2): In 15 tests, six statistically significant effects showed better treatment group outcomes, relative to the control group, for 1) survey-reported outcomes of any employment, current employment, earning $10/hour or more, and current average earnings, and 2) administrative outcomes of employed in year 2 and earnings in year 1. The administrative results were weaker than the survey results, which may be explained by lack of self-employment and official job data.

Youth Development (Table 3.3): In eight tests, three statistically significant effects showed better treatment group outcomes, relative to the control group, for survey-reported outcomes of civic engagement (volunteered, voted, or involved in local politics). The program had no consistent or lasting effect on measures of self-esteem, self-confidence, depression and happiness, or orientation toward the future (see Appendix C).

Delinquency and Risky Behavior (Table 3.4): In 13 tests, there were no statistically significant effects.

Checks for moderation showed that program effects on work and earnings tended to be larger for less-educated young people, although the differences were not always statistically significant.

A cost-benefit analysis found that the program benefits through four years did not outweigh the costs, but the authors suggested that benefits may continue to accrue over the participants' lifetime.