Big Brothers Big Sisters of America

Study 1

Summary

Tierney et al. (1995), Grossman and Tierney (1998), Grossman and Rhodes (2002), and Rhodes et al. (1999)  used a randomized controlled trial to assign 1,138 youths to a treatment group or a waitlist control group. Assessments at baseline and 18 months after baseline included a wide range of outcomes, including initiation of substance use, academic performance, relationships with family and peers, self-concept, and social and cultural enrichment.

Tierney et al. (1995), Grossman and Tierney (1998), Grossman and Rhodes (2002), and Rhodes et al. (1999) found that, at posttest, the intervention group relative to the control group had significantly

• Lower initiation of illicit drug use
• Fewer times of hitting someone
• Fewer skipped classes and days of school
• Better overall parental relationship.

Significant Program Effects on Risk and Protective Factors:

• Higher competency for schoolwork
• Higher parental trust and less lying to parents.

Evaluation Methodology

Design:

Recruitment: Sites for this study were selected from eight program offices nationwide which met the criteria for a large caseload and geographic diversity. The sites included: Philadelphia, Rochester, Minneapolis, Columbus, Wichita, Houston, San Antonio, and Phoenix. Those ages 10-16 were recruited from these sites over a 17-month period (October 1991 to February 1993). A total of 1,138 eligible youths joined the study.

Assignment: The study randomized youths to a treatment group (n = 571) that case managers attempted to match with mentors or to a control group that waited 18 months for a match (n = 567). Baseline assessments followed randomization but before informing participants of their condition. A total of 1,107 participants (97.3%) completed the baseline assessment, including 554 in the treatment group (97.0%) and 553 in the control group (97.5%).

Assessments/Attrition: The follow-up assessment occurred 18 months after baseline, from April 1993 to September 1994. The completion rate was 84.3% (n = 959).

Sample: The sample was just over 60% boys and over 55% minority (of minority youth, 71% were African American, 18% were Hispanic, 5% were biracial, 3% were Native American, and 3% were other racial/ethnic groups). Many in the sample came from poor households, with over 40% receiving either food stamps and/or cash public assistance. Household demographics included 90% living with one parent, and 5.6% living with only one of their grandparents (more common among minorities). One-fifth of the parents/guardians did not graduate from high school, and over 35% had completed only high school or earned a GED.

Measures: A total of 48 outcome measures of behaviors and social-psychological constructs across six outcome areas were used. Measures came from child self-reports. Alpha values ranged from .61 to .86 at baseline and from .61 to .90 at the follow-up. Many values fell below .70, but the authors viewed reliabilities of .60 and above as acceptable.

The six outcome areas included antisocial behaviors (using the Self-Perception Profile for Children), academic performance (using the School Value Scale, grades), attitudes and behaviors, family and peer relationships (using four scales from the Inventory of Parent and Peer Attachment; the Features of Children's Friendship scales were also used for peer relationships), self-concept (using the Self-Image Questionnaire for Young Adolescents), and social and cultural enrichment behaviors.

Another evaluation (Grossman & Rhodes, 2002) measured the impact of duration in mentoring relationships and identified predictors of early termination. The evaluation included measures of parent relationships, scholastic competence, grades and attendance, school value, self-worth, quality of relationship, and length of relationship.

Analysis: The analyses used multivariate linear and logistic regression analyses with numerous covariates, including the baseline outcomes (see Table A5 in Tierney et al., 1995, p. 47). The analyses of initiation of illegal drugs and alcohol used only those youths who at baseline had never used the substance, and the models did not include the baseline outcome control.

For the Grossman and Rhodes (2002) evaluation, participants were categorized into four groups (dummy variables), depending on how long their matches lasted. Multivariate regression was then used, controlling for baseline levels of variables. To measure predictors of relationship length, proportional hazard rate analysis was used. The four factors examined were baseline characteristics of the youth, baseline characteristics of the adult, characteristics of the match, such as whether the pair was matched primarily because of similar interests or race, and the quality of the relationship.

Intent-to-Treat: The researchers attempted to interview every youth who completed a baseline survey, and the main impact estimates included treatment youths who were unmatched with a mentor and did not participate in the program. However, the study also eliminated 12 cases from the 971 participants (1.2%) with both baseline and follow-up data. The follow-up data revealed that these participants "had actually been ineligible at baseline or that their control status had been compromised" (Tierney et al., 1995, p. 45). Most importantly, 11 of the 12 were controls who had been matched with a mentor before or during the treatment period.

Outcomes

Implementation Fidelity

Of the 487 youths in the treatment group, 378 (78%) were matched with a mentor during the study period. At the time of the 18-month follow-up, 40% of the matches were no longer meeting. Among closed matches, the pairs met for an average of 9 months; among ongoing matches, the pairs met for an average of 12.9 months. Over 70% met with their mentor at least three times a month and approximately 45% met one or more times per week. An average meeting lasted 3.6 hours.

Baseline Equivalence:

Tierney et al. (1995) stated that, for what appears to be the randomized sample, there were no significant differences between the treatment and control groups on baseline outcomes, demographics, or descriptive characteristics. Table A.5 lists condition means for 15 baseline measures. Grossman & Tierney (1998) presented a table of a subset of condition means for the analysis sample. They stated (p. 423, footnote 7) more broadly for all measures that "Using a 90% level of confidence, we could not reject the hypothesis that the treatment and control groups were the same at baseline."

Differential Attrition:

Attrition rates from treatment and control conditions were similar (85.3% vs. 83.2%, respectively), but the study presented no tests for differences between completers and dropouts. The statements about baseline equivalence for the analysis sample suggest that attrition was not a problem, but again details were lacking.

Posttest:

Tierney et al., 1995; Grossman & Tierney, 1998: Of the 48 tests of outcome measures, eight showed significant treatment benefits (p < .05), all in three of the six domains. Relative to the control group, the intervention group showed significantly reduced initiation of illicit drug use and incidence of hitting someone (anti-social behavior domain), significantly increased perceived ability to complete schoolwork and reduced number of skipped classes and skipped days (academic domain), and significantly better overall parental relationship and parental trust and lower number of times lied to parents (parent relationship domain). Tables in the text list results for 26 outcomes, while 22 non-significant outcomes were relegated to Appendix B.

For each outcome, the tables presented intervention effects separately by gender and by race and gender combined. Only a few of the numerous tests for group differences in treatment effects reached statistical significance. Although the results could be due to chance given all the moderation tests, it appears that white girls in the treatment group had the largest reduction in skipped classes.

Grossman & Rhodes, 2002: Results indicate that termination of the mentor relationship within the first 3 months had a negative effect on global self-worth and their perceived scholastic competence. Those who remained in sustained relationships with their mentors for more than 12 months reported significant increases in their self-worth, perceived social acceptance, perceived scholastic competence, parental relationship quality, school value, and decreases in both alcohol and drug use.

When a Two Stage Least Squares method was applied, results indicate no significant positive effects for matches lasting less than six months, and a significant increase in the use of alcohol. In these analyses, the matches lasting three months were not separated from those lasting six months. In the 6-12 month group, a few significant academic and behavioral outcomes emerged. The largest number of significant, positive effects emerged from the 12-month or longer group in academic, psychosocial, and behavioral outcomes.

Moderation (Rhodes et al., 1999): In an analysis of moderation by foster parenting, the article selected a subsample of 90 children living with foster parents - both relative and non-relative foster parents - from the larger sample. It then matched the foster parent subsample to a group of 90 non-foster participants living with their mother, father, or both. The matching variables included several demographic variables such as gender, age, race, state of residence, and disability status. The sample of 180 ended up with 56% from the intervention group and 44% from the control group.

In a repeated measures analysis of variance, which allowed the effects of treatment to vary by foster parenting, the results revealed a significant three-way interaction across group, time, and foster parenting for an outcome measure of prosocial support from peers. The treatment proved significantly more effective for youths with relative foster parents but significantly less effective for those with non-relative foster parents.

Mediation (Rhodes et al., 2000): In a mediation analysis, the article specified a model in which mentoring 1) improved the quality of parental relationships, which increased global self-worth, the value of school, and perceived scholastic competence, and 2) improved the quality of parental relationships, global self-worth, the value of school, and perceived scholastic competence, which increased grades and reduced skipping school. The mediating and outcome measures were all assessed at the 18-month follow-up rather than having the mediating measures affect outcomes at a later time point.

The results showed significant indirect effects of mentoring on global self-worth, the value of school, perceived scholastic competence, skipping school, and grades.

Mediation and Moderation (Rhodes et al., 2005): In examining substance use, the article first dropped 31 participants from the original sample who were terminated early and rematched with another mentor (n = 928 rather than 958). The mediation model specified that mentoring improved parent relations, peer relations, and global self-worth, which reduced substance use. The model did not fit well for the full sample, however.

The moderation analysis came from examining the mediation model separately by length of relationship between the youth and mentor. For a subgroup of youths who were in longer-lasting relationships, the analysis found that parental relations mediated the effect of mentoring on substance use.

Study 2

Summary

Thompson and Kelly-Vance (2001) used a quasi-experimental design to examine 34 at-risk boys who volunteered for the study. The treatment participants had been previously matched with mentors, while the comparison participants were waiting to be assigned a mentor. A posttest assessment examined educational achievement.

Thompson and Kelly-Vance (2001) found at posttest that the treatment group had significantly higher scores than the comparison group on

• Composite academic achievement
• Reading
• Math.

Evaluation Methodology

Design: Participants in the treatment group were matched with a mentor after a stringent screening process, and then met with their mentor on average, two to four hours weekly, for a commitment of one year. Matches were supervised by case managers through contacts with the parent, youth, and mentor. Training was provided to all volunteers and families.

Sample: Participants of the quasi-experimental design study were recruited from the BBBS of the Midlands, a well-established agency. The rigors of screening and matching mentor pairs, and the support structure in place, promote successful relationships. Treatment youth were boys recruited from agency events, while control participants consisted of boys who had been accepted into the BBBS program but were waiting to be assigned a mentor (average length of time on list was 15 months). Control youth boys were recruited at program orientation meetings and through telephone calls. Recruitment continued until the control group had as many participants as the treatment group. Written parent and youth consent were received before the initial assessment. The original study contained 17 participants in each group. At post-test, 12 treatment youth and 13 control youth remained. Average age of treatment youth was 11.9, while the average age of control youth was 10.4. The sample was predominantly Caucasian (92% of treatment, 77% of control), with African American (15% of control group only) and Hispanic (8% of treatment, 8% of control) youth also represented. All study participants had the risk factor of being from a single parent home, and at least one additional risk factor in order to be eligible to participate. These risk factors included family, school, peer, and substance use risk factors.

Measures: Each participant was administered the Kaufman Test of Educational Achievement (K-TEA) Brief Form. This was a composite measure which also yielded scores for reading, math, and spelling. In order to control for the impact of differential cognitive ability on achievement, participants were also administered the Kaufman Brief Intelligence Test (K-BIT). Both tests were administered at pre-test, with the K-TEA also administered eight to nine months after the first administration. Mentored youth also indicated the amount of contact with their adult mentor.

Analysis: ANCOVA analysis was used to account for possible pre-existing differences in intellectual functioning levels between the two groups. Cognitive ability was used as a covariate in order to account for preexisting differences between the two groups.

Outcomes

Baseline Equivalence and Differential Attrition: Of the 34 youth participating at baseline, only 25 remained at posttest. There was neither mention of a baseline equivalence assessment nor an analysis of differential attrition.

Posttest: There was a significant impact of mentoring on composite scores for academic achievement. Adjusted mean scores in reading and math also indicated significant differences between the two groups, with no significant difference in spelling scores.

Study 3

Summary

Turner and Scherman (1996) used a quasi-experimental design to examine 45 boys in the state of Oklahoma. The treatment participants had been previously matched with mentors, while the comparison participants were waiting to be assigned a mentor. A posttest assessment examined self-concept and problem behavior.

Turner and Scherman (1996) found at posttest that, relative to the comparison group, the intervention group reported a significantly

• Improved self-concept.

Evaluation Methodology

Design: This study used a small sample (n=45) of boys involved with the BBBSA program in the state of Oklahoma and measured the impact of the program on self-concept and behavior.

Sample: The intervention group (n=23) consisted of boys aged 9-15 who had been matched with a big brother for at least 6 months, while the control group (n=22) consisted of boys aged 7-13 who were on a waiting list to be matched.

Measures: The Piers-Harris Children's Self-Concept Scale and the Child Behavior Checklist, Parent Version (CBCL) were used to measure the target variables. The self-concept scale is a self-report measure that clusters 6 variables (Behavior; Intellectual and School Status; Physical Appearance and Attributes; Anxiety; Popularity; and Happiness and Satisfaction), while the CBCL has parents self-report on their child's behavior for a composite Problem Scale. There are 8 areas (Withdrawn; Somatic Complaints; Anxious/Depressed; Social Problems; Thought Problems; Attention Problems; Delinquent Behavior; and Aggressive Behavior). The two instruments were mailed to participants with a consent form. Instructions indicated that participation in the study would not jeopardize their status within the BBBS agency. Of the 92 pairs of instruments mailed, 45 usable pairs (48%) were returned.

Analysis: An analysis of means and standard deviations on the ratings on both scales was conducted.

Outcomes

Baseline Equivalence and Differential Attrition: This study did not indicate the demographic make-up of the sample, nor mention baseline equivalence or differential attrition analysis.

Post-test: Results indicate that boys who were matched with a mentor reported higher self-concepts than those who had yet to be matched. Further analysis of four selected subscales indicated that intervention boys reported significantly higher ratings of their physical appearance and popularity, and significantly less feelings of anxiety than boys in the control group. There were no statistically significant differences found with regard to mothers' ratings on the CBCL, although trends in the results did favor the matched boys.

Study 4

Summary

DuBois and Neville (1997) used a quasi-experimental study to examine 67 mentors at a single university. Mentors in the BBBS program served as a treatment group, and mentors in a service-learning course served as the comparison group. A six-month assessment measured mentor ratings of their relationships with their mentees.

DuBois and Neville (1997) found that, relative to comparison mentors, treatment mentors reported

• More benefits for their mentored youths.

Evaluation Methodology

Design: The responses of BBBS mentors were compared to those partaking in a mentoring program through a university service-learning course in the same city.

Sample: Participants in the sample included 27 BBBS mentors (69% of the 39 volunteers at the agency) and 40 university student participants. Of the BBBS mentors, 12 were male and 15 were female, with an average age of 28.78 years. The majority were White, with the exception of 2 African American women. The youth they were mentoring had an average age of 11.57 years, and included 2 African Americans and 21 Whites (background information was not available for 4 of the youth). University students consisted of 12 males and 28 females whose average age was 20.62 years. Again, the majority were White. Youth mentored by the university students had an average age of 15.37 years, with the majority being White (n=30), with the remainder representing various backgrounds, including African American (n=4) and Chicano and Latino (n=4). All youth in both programs were the same sex as their mentors.

Measures: Data was collected via questionnaire, collected from BBBS participants once a month for a 6-month period, and from the university students, once approximately 12 weeks into their semester. The BBBS volunteers assessed a variety of characteristics of the mentoring relationship, including the amount of youth-mentor contact, subjective feelings of closeness toward the youth, obstacles in the relationship, degree of contact with agency staff, and the frequency with which the volunteer and youth discussed various topics or engaged in different types of activities. In the final month, mentors were also asked to rate the extent to which youth benefited from the relationship during the 6-month period. The university students completed one survey with generally similar questions, with the most notable difference being that resulting measures of relationship characteristics (e.g., amount of mentor-youth contact) were based on only the prior month.

Analysis: Zero order correlations were made to assess differences in relationship characteristics and perceived benefits between Big Brothers/Big Sister mentors and university students. Tests were two-tailed, but significance was set at p<=.10. Analysis was intent-to-treat and did include all youth regardless of level of exposure.

Outcomes

Post-test: Results indicate that the majority of BBBS volunteers (82.6%) felt youth received "moderate" benefits from the relationship over the 6-month period, with the remainder reporting "small benefit." The university sample felt youth received "great" (20%) or "moderate" (40%) benefit, with nearly all remaining reporting "small" benefit. There was an inverse relationship found between the length of time the BBBS mentor had been in a relationship and the amount of contact they had with agency staff. BBBS mentors who had been in longer relationships also, however, reported less extensive contact with youth and fewer relationship obstacles. These associations were similar among the university students. With regard to relationship characteristics, in the BBBS sample, reports of mentor-youth contact and closeness were positively associated with perceived benefits for youth, whereas reports of contact with agency staff were negatively associated with perceived benefits. Length of relationship, when controlling for the reported level of average monthly contact, yielded a strong significant positive association with perceived benefits. For the university students, there was again a positive association with reports of mentor-youth contact and closeness on perceived benefits. However, there was no significant association found between length of relationship and benefit ratings, even when controlling for rate of contact. This may have been the result of the shorter length of relationship in this program (1-7 months) compared to the BBBS relationships.

Study 5

Summary

De Wit et al. (2006) used a randomized controlled trial to examine 71 youths in two Ontario, Canada, agencies. The treatment group received mentors, while the control group had to wait 12 months. A one-year assessment measured numerous outcomes relating to problem behavior, mental health, academics, and positive relationships.

De Wit et al. (2006) found no significant program effects.

Evaluation Methodology

Design: Two agencies in Ontario recruited families over a 12-month period. Families who were eligible for receiving a mentor for their child must have had a child between ages 7 and 14. Only one child per family was eligible to participate, so if there were multiple children in an eligible family, one was randomly selected to participate. Of 72 targeted families, 71 agreed to participate. The 71 families were randomly assigned to either the treatment group or a waitlisted control group, who would receive the program after at least 12 months. There were 39 treatment children and 32 controls. Waitlist controls were able to participate in recreational and educational activities. Assessments were conducted with children by agency staff at baseline and 12-month follow-up, while parents and mentors completed self-administered surveys on the same assessment schedule.

Sample Characteristics: Demographic data on the groups were as follows: 45% of children were between the ages of 7 and 9 years; 51% were boys and 77% came from a single parent family, 72% of which were female-headed households; 24% of children had caregivers with less than a high school education, 51% of families had a gross household income of under $20,000, and 36% received government assistance; 35% of the children were of a minority ethnicity (African Canadian, Aboriginal, Asian, Hispanic, Arab, or Jewish); 94% of parents enrolled in the program were female and these parents were, on average, 43 years of age; 60% of parents were currently divorced, separated, or widowed, while 45% reported a long-term physical or mental health problem. The majority of mentors (54%) were male and of a mean age of 27 years, and the majority, like the majority of youth, were white European.

Measures: The Strengths and Difficulties Questionnaire (SDQ) was used to measure child- and parent-reports of externalizing and internalizing problems, specifically emotional problems, conduct problems, hyperactivity and inattention. To assess children's behavior problems at school, children reported how frequently (from "never" to "four or more times") they either experienced or engaged in truancy, disciplinary referrals, delinquency, and disruptive behavior. Parents also provided some information for this domain, reporting the number of times their child got into trouble for misbehaving in school. Indirect aggression, or children's attempts to harm peers through manipulation, was assessed with items from the National Longitudinal Survey of Children and Youth. The Centre for Epidemiology Studies Depression Scale (CES-D) was used to measure frequency of depressive symptoms in the past week. Social anxiety was assessed with the Revised Social Anxiety Scale for Children (SASC-R), looking specifically at fear of negative peer evaluations, social avoidance and distress specific to new situations, and social avoidance in general. Academic achievement was measured with child- and parent-reports of letter grades received and English, science and mathematics performance. Children and parents were also both asked to report on child's community involvement and how frequently children participated in various activities. The HARE self-esteem scale was used to measure peer self-esteem and self-image, and the Survey of Children's Social Support assessed children's perceptions of how often they received support from a significant other, allowing for peer, teacher, and parent social supports. Quality of children's relationships was assessed for parent-child relationships (by parents), teacher-child relationships (child and parent reports), and friend-child relationships (child and parent reports). Children's social skills were measured with child- and parent-reports on the Elementary Level Student and Parent Forms of the Social Skills Rating System (SSRS), while children provided reports of coping skills with the Coping Scale for Children and Youth (CSCY). More specifically, these two instruments provided measures of cooperation, empathy, self-control, assertion, responsibility, cognitive behavioral problem solving, cognitive avoidance, behavioral avoidance, and assistance seeking. Other measures included child- and parent-reported attachment to school and child-reported sense of school safety and bullying.

Analysis: Data analysis was conducted with repeated measures analysis of covariance (ANCOVA).

Outcomes

Baseline Equivalence and Differential Attrition: 59 of the 71 families completed post-test assessments for an overall attrition rate of 18%. Of the 33 treatment group children who were assigned to receive a mentor during the 12-month period, 26 were matched, indicating that 21% of those who should have participated in the intervention with a mentor did not. At baseline, groups significantly differed on 5 of 45 outcome measures - indirect child aggression, academic self-efficacy, teacher social supports, self-reported physical attractiveness, and quality of parent/child relationship as reported by parents. There was also evidence of differential attrition by experimental group on 3 of the 45 measures, such that those who dropped out of the study had significantly higher scores on school misbehavior and significantly lower scores on assertiveness and attachment to school than those who were retained.

Posttest: Analysis revealed only marginally significant program effects on 5 of the 45 outcome measures, all from child self-reports. Program youth made improvements, relative to controls, on emotional problems (p = .08), fear of negative peer evaluation and generalized social anxiety and distress (in both cases p = .10), social support from teacher (p = .07), and self-control skills (p = .08).

Study 6

Summary

Thompson (1999) used a quasi-experimental design to examine 34 boys in Omaha, Nebraska, who had been assigned a program mentor (the intervention group) or were waiting for assignment of a mentor (the comparison group). A posttest eight to nine months after baseline assessed academic performance.

Thompson (1999) found that, relative to the comparison group, the intervention group scored significantly higher at posttest on

• Academic achievement in reading, math, and overall.

Evaluation Methodology

Design:

Recruitment: The study recruited 34 boys in Omaha, Nebraska, who were participating or waiting to participate in one local program. All participants came from a single-parent home and had an additional risk factor such as truancy, poverty, or substance use.

Assignment: In this quasi-experimental design, the treatment group consisted of boys who had a mentor, and the control group consisted of boys who had been accepted into the program but were waiting to be assigned a mentor. The controls had been on the waiting list an average of fifteen months. The author did not describe the basis on which some had received mentors and others had not.

Assessments/Attrition: The posttest came eight to nine months after the baseline assessment. The completion rate at posttest was 74%.

Sample:

Ages ranged from 7.6 to 15.9 years. The ethnic distribution was 92% Caucasian and 8% Hispanic in the treatment group, and 77% Caucasian, 15% African American, and 8% Hispanic in the control group.

Measures:

The academic performance outcome measures came from the Kaufman Test of Educational Achievement Brief Form, a normed and standardized test that the youth participants completed. The instrument measured the total score and three subtests for reading, mathematics, and spelling. The reported alpha of .85 came from another study.

Analysis:

The analysis of covariance included a cognitive ability measure as a covariate, but neither the description nor the tables were clear enough to determine if a baseline outcome was also used as a covariate.

Intent-to-Treat: The study dropped one treatment participant for low participation in the program and an unspecified number of treatment participants whose mentoring relationships were discontinued voluntarily by the mentor and youth.

Outcomes

Implementation Fidelity:

Not examined.

Baseline Equivalence:

The study presented no tests, but the ethnic distribution appeared to vary considerably across conditions (e.g., 92% versus 77% Caucasian).

Differential Attrition:

Attrition rates were similar across conditions (71% versus 76%), but the study presented no tests for differential attrition.

Posttest:

According to Table 3, the Group predictor significantly affected one of four outcomes (math). The Group x Outcome predictor significantly affected three of four outcomes (total, reading, and math). The author interpreted the latter coefficients as the key tests of the program.

Long-Term:

Not examined.

Study 7

Big Brothers Big Sisters of Ireland is run by Foróige, a national youth organization that adopted the core components of the Big Brothers Big Sisters program. According to program guidelines, "Project officers are expected to operate the programme in strict accordance with the BBBS Service Delivery Manual."

Summary

Dolan et al. (2010) used a randomized controlled trial to examine 164 youths ages 10-14 living in western Ireland. Study participants were randomly assigned to an intervention group that received mentoring along with usual organizational services or a control group that for two years received only the organizational services. Assessments 22 months after baseline measured substance use, conduct problems, and academic achievement.

Dolan et al. (2010) found that, relative to the control group, the intervention group had significantly better

• Prosocial behavior at the last assessment only (parent-rated)
• Scores on self-reported hope across all assessments
• Self-reported overall support and non-parental adult support across all assessments.

Evaluation Methodology

Design:

Recruitment: The sample came from new referrals (aged 10-14 years) to the program in five counties in Western Ireland from summer 2007 to February 2008. Referrals came from other agencies, typically for youths at low to medium risk. The search for a sufficient number of participants took longer than anticipated and about one in three young people declined to participate. The final sample size was 164.

Assignment: Random allocation was blocked by gender and location, with 84 assigned to the intervention group and 80 to the control group. The intervention group received the usual Foróige services plus the mentoring intervention, while control group participants received only the Foróige services until October 2009, when they were offered a mentor.

Assessments/Attrition: Survey assessments of youths, parents, and teachers occurred at four time points: baseline (December 2007), 10 months (October 2008), 17 months (May 2009), and 22 months (October 2009). At the last assessment, the response rate was 82% for young people, 79% for parents, and 6% for teachers. The program expects that matches will last for a minimum of one year and typically last longer. The 22-month assessment therefore would not be considered long-term.

Sample:

The overall sample consisted mostly of Irish-born youths (87%), with 7% from a Traveller background. The gender breakdown was 49% male and 51% female, and the average age was 12 years. Most young people lived in or near an urban location. About 46% of the sample did not live with both parents.

Measures:

The main outcomes came from youth self-reports. The outcome domains included emotional well-being (Hope Scale, Social Acceptance Scale), educational outcomes (liking school, scholastic efficacy, plans for college), risk behavior, and relationships and social support. Behavioral outcomes within the risk behavior domain included misconduct, alcohol use, and cannabis use. Scale reliabilities were generally good, with seven of 56 falling below .70 and only one below .60 (Table 9).

Parents reported on five subscale measures from the Strengths and Difficulties Questionnaire; 11 of 20 alphas fell below .70 and two fell below .60 (Table 55). Parents also rated their child's academic achievement.

Analysis:

The analysis used multilevel models for repeated measures that included outcomes in waves 2, 3, and 4 plus a baseline outcome control. Because of variability in the timing of the interviews, time in months from baseline to each follow-up assessment varied across individuals. The parent measures, however, lacked information on the exact timing, and the model used the wave number as a measure of time. The models included a time-by-group interaction that reflected condition differences in change from wave 2 to wave 4. With a non-significant interaction term, the authors interpreted the group effect as the average difference between the two conditions across the three follow-up waves.

Intent-to-Treat: The analysis used multiple imputation to include 144 of the 164 randomized participants (88%). Those excluded lacked baseline data or had no follow-up data and were deemed inappropriate for imputation.

Outcomes

Implementation Fidelity:

Of the 84 members in the intervention group, 72 (86%) were matched with a mentor. A total of 75% of matches were still ongoing at the last time point in the survey (October 2009). The authors noted that the program as implemented fell short of its ideal standard. Only 57% of the 72 young people matched with a mentor were matched for 12 months or more during the study period. At least 21% of matches ended before 12 months, including 7% that ended before six months. On average, working with a mentor began 6.48 months following the baseline interviews. The average duration that participants in the intervention group worked with a mentor was 11.79 months from the time they were matched with a mentor to the time of their final interview.

Baseline Equivalence:

For the student surveys, the authors reported that no significant baseline condition differences were found for the outcome measures. The study did not report on tests for sociodemographic measures.

For the parent surveys, two baseline measures of child emotional symptoms and conduct problems were significantly higher in the intervention group than the control group.

Differential Attrition:

For the student surveys, the authors stated that "A small number of people withdrew following the randomisation process since they objected to their status (i.e., they were not chosen for the intervention group, or else their friends were not and they were)." Condition differences in missing data were not significant, however. Comparisons of baseline outcomes, age, and gender (about 16 measures overall) between participants with complete data (i.e., all four interviews) and participants without complete data found two significant differences. Those missing data had, unexpectedly, higher scores on School Liking and Total Social Support.

For the parent surveys, the control group had significantly more missing data than the intervention group. The authors said that control parents may have been "more prone to dropping out if their child was not going to receive the intervention (i.e., mentoring)."

Posttest:

For the three youth-reported behavioral outcomes of misconduct, alcohol use, and cannabis use, there were no significant group or time-by-group effects. For the seven parent-reported behavioral outcomes, there were no significant main effects and one significant group-by-time effect on prosocial behavior. Graphs showed that the intervention group had higher scores only at the last assessment (d = .28).

For the 11 risk and protective factors, there were three significant group effects and no significant time-by-group effects. The intervention group reported higher scores than the control group on the Hope Scale (d = .22), social support from other adults (d = .13), and total social support (d = .13). Note that one of these three outcomes showed evidence of differential attrition.

Moderation tests found greater program benefits for youths not living with both parents.

Long-Term:

Not examined.

Study 8

Herrera et al. (2013) examined youths from seven mentoring programs, five of which were part of Big Brothers Big Sisters (BBBS) agencies. Random assignment of youths was conducted in only two of the agencies, both part of BBBS. Only the randomized design for the two agencies is described below. The results for agencies relying on non-randomized designs had the problem of mixing outcomes for the other three BBBS agencies with the two non-BBBS agencies. Herrera et al. (2022) focused their report on the two randomized agencies only.

The two BBBS agencies participating in the randomized design were required to make several program enhancements. These included: 1) early mentor training, 2) ongoing match support via contact with mentor, youth, and parent/guardian, 3) mentor-youth meetings three or more times a month, and 4) a commitment of 18 months for mentor and youth. These enhancements line-up closely enough with the normal program requirements to include as part of the BBBS program. However, the two agencies may not be typical of BBBS agencies across the country.

Summary

Herrera et al. (2013, 2022) used a randomized controlled trial to examine 764 youths in two Washington State program agencies. Study participants were randomly assigned within the agencies to the intervention group or a waitlist control group. Assessments covering outcomes such as depression, grades, misconduct, and prosocial behavior occurred at baseline and a 13-month posttest.

Herrera et al. (2013, 2022) found that, relative to the control group, the randomized intervention group reported significantly

• Lower depression (a primary outcome)
• More outcomes showing positive change (a primary outcome)
• Fewer parent-reported emotional symptoms (a secondary outcome)
• Better relationships with non-parent adults (a secondary outcome)
• Fewer parent-reported conduct problems (a secondary outcome)
• Fewer parent-reported peer problems
• Fewer parent-reported total difficulties.

Evaluation Methodology

Design:

Recruitment: Two BBBS organizations in Washington State with active mentoring programs were selected for the study. Both could randomly assign youth to conditions and implemented the full set of program enhancements. Of the families asked to participate in the study over the one-year enrollment period (Spring 2008 to Spring 2009), only 1-2% refused. A total of 764 youths (ages 8-15) and their parents joined the randomized study. Although not required to be high risk, the youth participants faced more risk factors than average and were considered higher risk rather than high risk.

Assignment: For the two programs using randomized controlled trials, half of the youths were randomly selected to be matched immediately with mentors (treatment group, n = 379), while the remaining half (waitlist control group, n =385) were not eligible for matching until 13 months into the study. To ensure that youths were matched as soon as possible after completing their baseline, youths and parents often did not complete the baseline survey and were not randomized until a mentor (or potential mentor) had been located. Note that the randomized part of the study included only participants who enrolled in the first year (2008) of the two-year study.

Herrera et al. (2022) noted that 199 of the participants enrolled in the study with one or more siblings. In these cases, siblings were randomly assigned together, as one unit, to join the same condition.

Assessments/Attrition: A posttest assessment came 13 months after the baseline assessment. Herrera et al. (2022) reported a completion rate of 85.6%.

Sample:

Figures for the analysis sample in Table B.1 (Herrera et al., 2013) showed an average age of about 11.4. Males made up 58% of the youths. The racial and ethnic composition of the sample was 40% White, 28% Black, 21% Hispanic, and 11% others.

Measures:

Herrera et al. (2013) examined 21 youth-reported and three parent-reported outcomes falling into eight domains. The primary outcomes consisted of eight pre-selected measures, one from each domain, and all came from youth self-reports. The eight measures and domains covered outcomes such as depression, grades, misconduct, and prosocial behavior. Two additional primary outcomes measured the number of positive reliable changes experienced by an individual over the 13-month study period and the number of negative reliable changes experienced over the same period. The 14 secondary outcomes were presented only as a supplement in the appendix. For all outcome measures, alpha values reached or approached acceptable levels, ranging from .66 to .88 at baseline and from .67 to .89 at follow-up.

Herrera et al. (2022) examined eight youth-reported outcomes and six parent-reported outcomes (all pre-specified) that were similar but not identical to those in Herrera et al. (2013). To assess overall program impact, the authors created a composite measure by combining 13 of the 14 youth- and parent-reported outcomes. Some of the measures had only marginally acceptable reliabilities.

Analysis:

The individual-level analyses used linear or logistic regression models, depending on the outcome, with controls for the baseline outcome, the agency, the survey form (in-person or by phone), key background characteristics, and any measures that differed (p < .10) between conditions at baseline. The analyses for three outcomes (i.e., substance use, legal problems, and skipping school) were more limited, however. Only those who reported never having engaged in the behavior at baseline were included and the baseline outcomes were not used as a control.

Herrera et al. (2022) also checked for the non-independence of siblings within families by estimating mixed models with family as a random intercept.

Intent-to-Treat: The complete-case analyses included participants with data at both baseline and posttest, regardless of the mentoring they received. The study used mean replacement plus missingness dummy variables for the small amount of missing data on baseline measures but did not impute missing follow-up data or use FIML. Herrera et al. (2013) dropped some participants from the analysis when they "illogically" reported at baseline having engaged in a given behavior (for example, skipping school or having problems with the law), but then reported at follow-up that they had never exhibited these behaviors. These participants were excluded from analyses examining the specific outcome measure for which their response was illogical. Herrera et al. (2022) used complete case analysis with exclusion of those missing follow-up outcomes but made no mention of other exclusions.

Outcomes

Implementation Fidelity:

Almost three quarters of the sample reported at least a moderately positive relationship with their mentor. Mentors, on average, also reported fairly strong feelings of closeness toward their mentee. However, the average length of the first match at follow-up was about 8.95 months, less than the desired 13 months, and only about 60-67% of participants were in an active match at follow-up. Herrera et al. (2022) also noted that 19 youth were never matched with a mentor.

Baseline Equivalence:

Herrera et al. (2013) used the analysis sample to conduct 25 tests for equivalence between the randomized treatment and control groups (Table B.1). They found three statistical differences at p < .10. The treatment group reported higher honesty, fewer legal problems, and more time spent on homework. A joint test across all 25 measures also reached significance. Note that the tests for condition differences controlled for youth age, race/ethnicity, mode of survey administration (phone or in-person), and agency.

Herrera et al. (2022) also used the analysis sample but reported tests without the controls (see Table 1). Of the 21 tests, none showed significant condition differences at p < .05. Three (i.e., race/ethnicity, free/reduced-price lunch status, and skipping school) showed marginally significant differences (p < .10), which were good-sized (e.g., 31.3% African Americans in the control group versus 23.8% in the treatment group).

Differential Attrition:

Herrera et al. (2013) reported multiple tests. First, the attrition rates for the treatment group and the control group (19% versus 17%) did not differ significantly. Second, comparisons of baseline characteristics for completers and dropouts revealed that those without follow-up data were older, more likely to be Hispanic, less likely to be white, reported fewer depressive symptoms, felt higher levels of social acceptance, and were more likely to have reported misconduct and having ever skipped school at baseline. Third, tests for differential attrition - defined as different patterns of attriter versus non-attriter differences in the treatment group compared to the control group - "found no evidence of this when conducting joint tests across all baseline characteristics (p = .26 for treatment versus control/comparison group)." Fourth, the baseline differences found for the analysis sample may reflect attrition.

Herrera et al. (2022) reported tests that compared those with and without 13-month survey data. The results revealed three statistically significant differences: Those without follow-up data were less likely to be from single-parent households (35.9% vs. 53.1%), reported higher levels of depressive symptoms (d = .21), and were more likely to report having skipped school (25.9% vs. 11.2%).

Posttest:

Of 10 primary outcomes tested (Table 5.2), two were significant. The randomized intervention group, relative to the randomized control group, reported significantly lower depression (d = -.14) and significantly more outcomes showing positive change (d = .15). When adjusting for multiple tests with the Benjamini-Hochberg factor, these two effects remained significant but for a one-tailed test.

Tests for all 24 primary and secondary outcomes (Table B.3) found three additional significant effects beyond the two significant primary outcomes. The three significant secondary outcomes were parent-reported emotional symptoms, relationships with non-parent adults, and parent-reported conduct problems.

Tests for moderation were reported, but they combined the randomized and non-randomized intervention groups and the BBBS and non-BBBS programs.

Herrera et al. (2022) found a significant condition difference for one of eight youth-reported outcomes: The intervention group reported fewer depressive symptoms (d = .146) than the control group. They also found a significant condition difference favoring the intervention group for four of six parent-reported outcomes: emotional symptoms, peer problems, conduct problems, and total difficulties (d values ranged from .138 to .253). Lastly, they found that the composite scale combining 13 of the 14 measures showed a significant condition difference favoring the intervention group (d = .213). Checks that accounted for non-independent clustering of siblings within families did not change the findings.

Long-Term:

Not examined.

Study 9

This study used the sample and baseline data from Study 1, but unlike the original randomized controlled trial, it relied on a quasi-experimental design because the long-term follow-up included participants in the waitlisted control group who had access to the program after the end of the original 18-month study.

The report appears to be preliminary, as the authors mentioned that they were still gathering additional data and planning additional analyses.

Summary

DuBois et al. (2018) used a quasi-experimental design to examine long-term outcomes for the sample used in Study 1, namely, 1,138 youths from eight program sites across the country. Given that the original control group received access to the program after the initial follow-up period, the design defined those who received a full year of mentoring or a full year of high-quality mentoring as the treatment group and others as the control group. In examining outcomes relating to education, crime, and well-being some 20 years after the initial study, the analysis used regression models to control for condition differences.

DuBois et al. (2018) found that, relative to the non-randomized group of participants that did not receive one year of mentoring, participants who did receive one year of mentoring had significantly

• Fewer criminal offenses.

They also found that, relative to the non-randomized group of participants that did not receive one year of high-quality mentoring, participants who did receive one year of high-quality mentoring reported significantly

• Less crime
• Lower substance use
• Better emotional, psychological, and social well-being.

Evaluation Methodology

Design:

Recruitment: The original sample from Study 1 (Tierney et al., 1995) included 1,138 youths ages 10-14 from eight program sites across the country. All age-eligible youths who applied for the program at the sites from October 1991 to February 1993 were recruited.

Assignment: Rather than relying on the original random assignment, the study used the actual mentoring received to define conditions. The design controlled statistically for baseline differences between non-randomized groups of participants who received and did not receive mentoring from BBBS.

Two measures of mentoring defined the conditions, one for the full sample and one for a subsample of adult survey respondents. The first measure for the full sample defined those receiving a full year of mentoring as the treatment group and others as the control group. The measure came from multiple sources, including the original study dataset, program records, parents, and adult surveys of the participants. For the 40% without match history information from any of the data sources, the mentoring measure was derived from a Decision Tree analysis. The procedure used all variables from the baseline assessment, including the randomly assigned condition and the relationship of the baseline variables to the mentoring outcome for the other 60% of the sample to predict (or impute) condition membership. About one-third of the sample met the one-year criteria (32.7%), which included 56.7% of the original treatment group and 8.5% of the original control group.

The second measure from the adult survey subsample used retrospective self-reports to define those having both a full year of mentoring and a close mentor relationship as the treatment group and others as the control group. About 21% of the subsample met these criteria.

Assessments/Attrition: The study did not precisely define the follow-up period but appeared to examine outcomes roughly 20 years after the start of the original study. For the educational outcomes, data were available for all 1,138 of the original participants. For the crime outcomes, data were available for 1,051 (92%) of the original participants. For the outcomes obtained from an adult survey, data were available for 290 (25.5%) of the original participants.

Sample:

At baseline, the youths were 63.7% male with an average age of 12.29 years. About 59.7% were minority (non-White). Nearly half (42.7%) lived in families receiving public assistance.

Measures:

The outcome measures, all from independent sources, came from records for the full sample and from self-reports for the adult survey sample. The measures included:

• Post-secondary education (records for the full sample) obtained from matches found in the Research Center of the National Student Clearinghouse; coverage rates were often low for the time when the sample would have attended college (range = .119 to .895).
• Adult arrests (records for the full sample) obtained from searches of state criminal records using Accurint; the coverage likely varied across states and over time, and the measures did not include records of federal arrests.
• Numerous measures of crime, employment, problem behavior, socio-emotional well-being, health, and substance use from the adult survey; alpha values for scaled measures were generally good, with two below .70.

Analysis:

The analyses used linear, logistic, and negative binomial regressions depending on the outcome measure. To assess the effects of non-randomly assigned conditions on the outcomes, the analysis relied on two methods. First, the regression models controlled for 57 baseline measures. The baseline controls overlapped with but did not exactly match the outcomes. The authors noted that for the analysis of survey measures, which had a much smaller sample size, the models pared down the number of covariates. Second, among those whose values on mentoring were imputed, the analyses were weighted based on the estimated probability of the designation provided for each of these participants.

Intent-to-Treat: In addition to the primary QED analysis in which participants were assigned on the amount of treatment received, supplementary intent-to-treat analyses included participants in their originally assigned condition.

Outcomes

Implementation Fidelity:

The study measured fidelity only in terms of the percentage of all participants who received a full year of mentoring (32.7%) and received a full year of high-quality mentoring (21.0%).

Baseline Equivalence:

The study adjusted for baseline differences without any tests or descriptions of the size of the differences.

Differential Attrition:

The adult survey sample included a significantly higher proportion of female participants and Whites and a significantly lower proportion of persons identified as having a learning disability and repeating a grade in school. The adult survey respondents also had moved significantly less often than others in the original study sample, reported lower global self-esteem, and were less likely to have been rated by case managers as likely to benefit from BBBS mentoring.

Posttest:

Not examined.

Long-Term:

Results for the full sample (see Table 1 and the text) showed that receiving a full year of mentoring significantly improved one of five outcomes: The total number of offenses was significantly lower for youths receiving a full year of mentoring than for others.

Results for the subgroup that completed the adult survey showed that receiving a full year of high-quality mentoring significantly improved seven of 17 measures. The high-quality mentoring group relative to others reported significantly fewer juvenile arrests (OR = .11), less adult stealing (d = -.40), and lower adult alcohol use (d = -.34), as well as significantly higher grit (d = .29), emotional well-being (d = .47), psychological well-being (d = .47), and social well-being (d = .48).

The intent-to-treat replication based on the originally assigned conditions (regardless of actual mentoring received) found no significant main effects (p < .05). However, the authors noted that the control participants who eventually received the treatment biased the estimates of program effects downward.

The study tested for a large number of interactions of mentoring status by numerous demographic variables. The tests found significant effects of mentoring on a few outcomes for racial/ethnic minority group members, participants with low parental education, and females. However, in a few instances, mentoring status also predicted poorer outcomes for particular subgroups. The authors noted that, given all the tests, the relatively few effects, and the lack of consistent direction, the subgroup results warranted caution.

Study 10

Summary

ICF International (2011) used a randomized controlled trial to examine 272 children ages 7-13 who had at least one parent in prison, on probation, or on parole. The study did not include the highest-risk program participants but randomized the others to a treatment group or an 18-month waitlist control group. Assessments at baseline and six, 12, and 18 months measured academics, well-being, and closeness to adults.

ICF International (2011) found that the treatment group relative to the control group reported significantly

• Better parenting (six months only)
• Stronger feelings of connection to school, community, and family (12 and 18 months)
• Stronger presence of caring adults in their life (12 and 18 months)
• Higher self-worth (six and 18 months)
• Greater ability to make friends (18 months only)
• More positive sense of the future (six months only).

Evaluation Methodology

Despite the different name and a focus on children of incarcerated parents, the Amachi program is based on core BBBS mentoring standards and operates within BBBS organizations.

Design:

Recruitment: The study examined children and youths ages 7-13 who had enrolled in the program at three Texas sites between August 2008 and April 2010. The three sites (Abilene-Dallas, Austin, San Antonio) were all operated by BBBS and included about 80% of the Amachi matches in the state. Eligible children had one or both parents, or a biological family member, in prison, on probation, or on parole. However, children at the highest risk and with greatest needs, who could not ethically be assigned to the control group, were assigned mentors but excluded from the study. Also, only one sibling in multi-sibling families was included in the study (although the other siblings belonged to the same condition as the study sibling). With a consent rate of 33% and after excluding siblings, the final sample was 272.

Assignment: Children were randomly assigned by their birth dates to either the treatment group to be matched with a mentor within three months (n = 138) or the control group to be placed on the "ready to match" list and tracked for 18 months (n = 134). One design limitation resulted from collecting some baseline data up to three months post-enrollment. The delay may have incorporated some treatment effects into the baseline measures and weakened the estimated program effects.

Assessments/Attrition: Data collection occurred at baseline and six, 12, and 18 months. Attrition rates were 18.4% at six months and 48.9% at 18 months.

Sample:

The sample children averaged 10.5 years of age, and the majority were African American, male, and living with their mother.

Measures:

The 13 outcome measures came from child and parent surveys, were independent, and were grouped into three categories: Child-Family/Community Relationship, Child Well-Being, and Academic/School. The first two categories largely consisted of risk and protective factors, while the last category consisted of behavioral outcomes. The component scales listed in Appendix B showed nine with alpha values below .60.

Analysis:

The analysis tested for condition differences with analysis of covariance models. The covariates included outcomes that differed at baseline.

Intent-to-Treat: The complete case analyses, done separately for each of the three follow-up periods, used all participants with both baseline and follow-up data (and no imputation or FIML). The sample sizes were thus larger at six months than 12 and 18 months.

Outcomes

Implementation Fidelity:

The analysis of mentoring data found that 81% of the matches were viable at six months, but by 12 months post-enrollment, only 54% of the matches had continued.

Baseline Equivalence:

Chi-square tests done for the baseline sample indicated no significant condition differences on the five demographic variables. For the 13 outcome measures, four (closeness; connection to school, community, and family; caring adult other than parents/caregivers; and literacy) had significance levels that were smaller than .05 and/or effect sizes that were equal to or larger than .20.

Differential Attrition:

Condition attrition rates differed by 6.8% at six months (and met the WWC optimistic standard for acceptable attrition) and by 24.3% at 18 months (and did not meet the WWC standards). The authors reported "that there were no statistically significant differences between the samples with complete data and those with incomplete follow-up data (see Appendix D)." It appears from the Appendix table that a comparison of completers and dropouts was done within conditions, but the wording was unclear. In addition, there appeared to be some statistically non-significant evidence of differential attrition. For example, 6% fewer males completed the follow-ups in the treatment group, while 10% more males completed the follow-ups in the control group.

Posttest:

The authors designated (post hoc) the six-month results as primary because it was the period of most intense mentoring and lowest attrition. The 12- and 18-month results were considered exploratory.

Child-Family/Community Relationship Outcomes. The primary analysis at six months found significant program effects on two of five outcome measures, both related to parenting. However, neither effect emerged as significant in exploratory tests at 12 or 18 months. Two other outcomes, feelings of connection to school, community, and family and the presence of caring adults in their life, were significant at 12 and 18 months but not at six months.

Child Well-Being Outcomes. The primary analysis at six months found significant program effects on two of three outcome measures, one relating to feelings of self-worth and one relating to sense of future. In the exploratory analysis, the effect on self-worth was again significant at 18 months, while the ability to make friends emerged as significant at 18 months.

Academic/School-Related Outcomes. The results indicated no significant effects on the outcomes at any of the follow-up periods.

Long-Term:

Not examined.

Study 11

This study examined a treatment group that came from both the community- and school-based mentoring programs. Because the combination prevents separation of the effects from the two different programs, the study is included here as well as in the school-based program.

Summary

Peaslee and Teye (2019) used a retrospective, posttest-only quasi-experimental design with non-random assignment and propensity score matching. The sample of 150 youths came from a single agency. After matching, the analysis compared those who had been in a relationship with a program mentor for at least 12 months (the intervention group) with those who never received a mentor (the comparison group). A long-term follow-up assessed risky behavior, prosocial behaviors, depression, and academics.

Peaslee and Teye (2019) found that, relative to the matched comparison group, the intervention group had significantly

• Higher expectations of finishing college
• Higher participation in organized activities
• Lower cumulative absenteeism.

Evaluation Methodology

Design:

Recruitment: The study participants came from a single affiliate of Big Brothers Big Sisters of America that served two local school divisions: a small urban city and an adjacent rural/suburban county. The sample included 150 youths who had enrolled in the program at ages 6-11 and who were attending local area schools at ages 12-16. Most but not all enrollees had been matched with a mentor.

Assignment: The quasi-experimental posttest-only design compared those who had a mentor with those who had enrolled in the program but were never matched with a mentor.

The treatment group included all newly matched youths in either the school- or community-based program from February 2012 to November 2013. Those eligible for the study received a mentor during childhood (ages 6-11 years old), had a successful mentor relationship of at least 12 months, closed the match at least 12 months prior to the study, and were currently between 12 and 16 years old. Of the 93 who met the eligibility criteria, 43 were enrolled in the community-based program and 50 were enrolled in the school-based program.

Members of the comparison group were identified from historical agency waitlists from 2007 to 2011 - a different period than for the treatment group. The comparison group included 57 youths who were enrolled in local area schools and who had earlier been referred to the program but had been 1) removed from the waitlist (either because the child aged out of the program's service range or because the family had lost contact with the agency), or 2) on the waiting list at least 18 months. To increase the size of the comparison group, the authors included some who did not meet the treatment group eligibility requirement of being referred in childhood (ages 6-11). The authors also noted (p. 8) that the comparison group had characteristics making them "difficult to match."

The QED used propensity score matching to adjust for condition differences. The size of the matched samples depended on the outcome measures and the extent of missing data. There were either 40 or 35 in each condition (80 or 70 total).

Assessments/Attrition: The assessment came at least 12 months after a treatment match closed, and the average time since match closure was 30 months for the treatment group. Time from baseline was approximately 51-54 months for the treatment group and 56 months for the comparison group. There was no attrition for most outcomes in the retrospective study. Dropouts and movers were instead excluded in the selection of the sample, but no figures on the extent of the selection were available. For some educational measures, outcome data were available for 80-90% of the full sample of 150.

Sample:

The average age of the initial treatment sample (Table 2) was 9.1 years old. About one-third of the treatment participants were male, nearly half were Hispanic, 57% were from households receiving public assistance, and about half were living in the city.

Measures:

Outcome data came from two independent sources: the Youth Outcome Survey and school records. The survey, which participants completed in school, included measures of social acceptance, school competence, grades, future aspirations, parental trust, peers risky behavior, truancy, personal risky behaviors, prosocial behaviors, and depression. The school records included measures of grades, absenteeism, and disciplinary infractions. The study did not report on reliability or validity but referred to more details in Appendix B, which was not available with the paper or online.

Analysis:

The analysis compared means and proportions of the matched treatment and comparison groups. The propensity score matching used the one-to-one nearest neighbor method with no replacement in either group and calipers of .20-.25. The predictors in the propensity model included individual-level child demographics (age, gender, race/ethnicity), household risk factors (family living situation, public assistance, maternal education, parental unemployment and incarceration), and environmental risk factors (school district, school poverty). Given the retrospective design, it was not possible to use baseline outcomes as predictors.

Intent-to-Treat: The analysis excluded program participants who did not have a successful 12-month match from the sample.

Outcomes

Implementation Fidelity:

The average match length among treated youths in the community-based program was 29.76 months, but the figure excluded anyone whose match did not last 12 months.

Baseline Equivalence:

For the full sample, Table 2 showed two of 12 sociodemographic measures to differ across conditions before matching, and none after matching (though there was a 10% difference in eligibility for free or reduced-price lunch after matching). For the subsample with educational data, Table 9 showed three of 12 sociodemographic measures to differ significantly across conditions before matching and none after matching (though there was a 9% difference in percent male). In addition, for both the full sample and the subsample, the comparison group was older than the treatment group by .85 years, which appears to equal about .60 standard deviations.

Differential Attrition:

The posttest-only retrospective design had no attrition but was unable to report on differential selection into the study. For the educational outcomes, the percentage missing data was substantially greater in the comparison group (33.3%) than the treatment group (3.3%). However, the propensity score matching sought to equalize conditions on measured variables.

Posttest:

Not examined.

Long-Term:

Of the 36 tests, three outcomes showed significant effects (p < .05): The treatment group reported significantly higher expectations of finishing college (OR = 3.4), higher participation in organized activities (OR = .29), and cumulative absenteeism (d = .63) than the comparison group.

Subgroup effects showed stronger treatment benefits on grades for Whites and Blacks than for Hispanics and on absenteeism for girls than for boys.

Study 12

A four-year assessment using official arrest records will follow this interim report on the 18-month survey results.

Summary

DuBois et al. (2022) used a randomized controlled trial to examine 1358 youths from 17 agencies located across the country. Study participants were randomly assigned within the agencies using a 3:1 ratio to the intervention group or a waitlist control group. Assessments covering primary outcomes related to delinquency and secondary outcomes related to academics, personal resources, family, and mental health occurred at baseline and an 18-month follow-up.

DuBois et al. (2022) found for the primary outcomes that the intervention group relative to the control group had significantly lower likelihoods of

• Self-reported arrest
• Self-reported substance use.

For the secondary outcomes, the intervention group relative to the control group had significantly

• Lower aggressive behavior
• Better social skills, self-control, self-advocacy, and grit
• More hopeful future expectations
• Better family functioning and parenting
• Greater school engagement and college exploration.

Evaluation Methodology

Design:

Recruitment: The study randomly selected and invited 54 agencies to participate and 17 agreed. As four withdrew either before the start or in the early stages of the study, new agencies were added to maintain the sample of 17 agencies. The agencies were located across 13 states and in both urban and non-urban areas.

Eligible youths came to the agencies from February 2018 through February 2020 and met study inclusion criteria: ages 10 and older, no severe learning, cognitive or other intellectual disability, a parent who both spoke and read English or Spanish, and no previous agency mentoring. Based on a prior agreement, agencies exempted 32 youths with special needs from the study. Of 5,379 youths assessed for eligibility, 1,358 (25%) met the eligibility criteria and consented to participate (the consent rate was 76.5%).

Assignment: After giving consent and completing the baseline assessment, youths were randomly assigned using a 3:1 ratio to the treatment group (n = 1,012) or the control group (n = 346). A total of 418 youths entered the study with at least one sibling, and siblings in the same family received the same assignment. The treatment participants were immediately eligible for matching with a volunteer mentor. The control participants were not eligible for matching until the end of the four-year study but had the opportunity to join waitlist activities such as sporting events, gym programs, educational activities, and special events.

Following randomization, four participants were dropped from the control group and one from the treatment group due to ineligibility or parent withdrawal, leaving 1,353 youths (1,011 in the treatment group and 342 in the control group).

Assessments/Attrition: The follow-up assessment occurred 18 months after study enrollment. Surveys were collected from 87% of either the youth or the parent and 80% of both the youth and parent.

Sample:

The 17 participating agencies had been affiliated with BBBS from five to 106 years and were medium to large in size, serving from 48 to 3,165 youths annually. Most of the youths were male (63%), and the average age was 12 years. The sample included 31% Hispanics, 39% Blacks, and 24% Whites.

Measures:

The outcome measures came from surveys that both parents and children completed online (or sometimes on paper). Researchers who were not blind to condition guided the youths through the survey and read the questions aloud, but answers were made confidentially. The 35 pre-specified outcomes listed on page 22 and in Appendix 2 distinguished primary from secondary categories.

The primary outcomes included arrests, property-related delinquent behavior, violence-related delinquent behavior, and substance use. The text noted that the primary outcomes changed slightly from pre-registration to improve their measurement properties. The secondary outcomes were divided into one risk-factor category and four protective-factor categories: personal resources, social-contextual resources, mental health and well-being, and academic engagement and performance. The single-item measures (e.g., times arrested) did not have reliabilities but have been commonly used in the literature. All but a few of the multi-item measures had good reliabilities (listed in Appendix 2).

Analysis:

The analysis used generalized linear and nonlinear mixed-effects models with random intercepts to account for clustering due to nesting of youths within both sites and families (i.e., siblings). The specific form of the model depended on the distributions of the outcomes (i.e., binary or continuous). Pre-specified covariates included youth demographics, baseline outcome values, and offending history. Other measures that differed across conditions at baseline by .05 standard deviations or greater were also included. For the primary outcomes, the Benjamini-Hochberg family-wise adjustment for multiple tests was used.

Intent-to-Treat: The analysis used multiple imputation to include all randomized participants. The imputation was done separately for the control and treatment groups and for parents and children. It used baseline outcomes, planned covariates, and other pre-specified measures as predictors.

Outcomes

Implementation Fidelity:

By the 18-month follow-up, 35% of the treatment youths had not been matched with a mentor. Those who were matched reported receiving an average of about 11 months of mentoring. Although COVID-related disruptions in agency operations decreased the rate of matching, it also appeared that the pre-pandemic rate of matching was lower than in previous program evaluations.

Baseline Equivalence:

The analysis using the full randomized sample (Table 3) found three significant differences in 46 tests. Measures exhibiting significant differences had standardized difference values of .15-.17, with the treatment group having more single-parent families but lower goal-setting and involvement in organized youth activities than the control group. Many measures had standardized differences greater than .05 and therefore were included as controls in the analysis models, but there was no obvious pattern favoring one condition.

Differential Attrition:

Survey completion rates were similar for the treatment (86%) and control groups (89%). The authors stated that the overall and differential attrition rates were within the What Works Clearinghouse conservative attrition standards. No other tests were reported.

Posttest:

For the primary outcomes, two of four tests were significant: self-reported arrest in the past 18 months (d = -.51) and self-reported substance use in the past 18 months (d = -.37). The authors noted (p. 33) that the effect on arrest was sensitive to the imputation model used.

For the 31 secondary outcomes, 10 were significant, including one behavioral outcome (aggressive behavior, d = -.17) and nine risk and protective factors (social skills, self-control, grit, self-advocacy, family functioning, inconsistent parental discipline, hopeful future expectations, school engagement, college exploration). Controls for survey completion after the pandemic start did not change the results.

The authors noted that, given 35% of the treatment youths were unmatched, the intent-to-treat analysis underestimated the impact of program participation for those who were matched with a mentor.

Long-Term:

Not examined in this interim report.