Wyman’s Teen Outreach Program

Study 1

Summary

Allen and Philliber (2001) conducted a mixed randomized and quasi-experimental design study with 3,277 high school youth. Self-report assessments of pregnancy, course failure, and academic suspension were administered within the first few weeks of the beginning of the program and academic year, and again at program exit at the end of the academic year.

Allen and Philliber (2001) found, compared to the control group, participants in the intervention group experienced significantly lower risks of:

• Pregnancy
• Course failure
• Academic suspension

Evaluation Methodology

The motivation for this evaluation was to learn whether the Teen Outreach program had greater efficacy for those students who were more at-risk for problem behaviors than other students.

Design:

Recruitment: In this mix of a quasi-experimental and randomized study, data were collected from 3,277 youth in grades 9-12 from 60 sites over the course of 4 years. The actual years of the data collection were not reported. No information is provided as to how the 60 sites were selected.

Assignment: The intervention group included 1,673 students and the comparison group included 1,604 students. About 20% of the study youth were randomly assigned to either the Teen Outreach group or the comparison group. The remaining Teen Outreach youth chose to participate in the program by requesting it after hearing about it during school announcements or agreeing to participate after having been referred by teachers or counselors. The remaining comparison group youth were selected through one of two methods: (a) program participants nominated youth whom they thought would fill out the questionnaire the same way they did; or (b) facilitators or guidance counselors nominated youth whom they thought were from similar backgrounds and had similar levels of problems.

Assessments/Attrition: The baseline assessment came within the first few weeks of the academic year, while the posttest came at the end of the academic year. The intervention group experienced an overall attrition rate of 8.9% and the comparison group exhibited a rate of 7.8%.

Sample characteristics: Of the 3,277 sampled youth, the mean age was 16 years old, with the majority in 9th or 10th grades. The average mother's education level was slightly over a high school graduate, and slightly over half (about 55%) of the youth lived in two-parent homes. About 19% of the sample had prior academic suspensions and 32% had failed at least one course. About 8% had either caused or been pregnant in the past and about 3% were teen parents. The samples differed only slightly on gender and race profiles.

Measures:

The measures came from a self-report questionnaire that program youth completed in their Teen Outreach classroom and comparison group youth completed in study hall or some other school setting. Students were told that the information they provided would be kept strictly confidential, and that only identity codes would be included on their questionnaires. They were specifically reassured that none of their answers would be available to program facilitators or to other school officials, and that no data which in any way identified them would be reported.  

The outcome measures came from questions about three behaviors over the last academic year: 1) having gotten pregnant or caused a pregnancy, 2) having failed any course, and 3) having been suspended. Each outcome took a binary form of yes or no. According to the authors, prior research suggested that student reports on these outcomes were generally accurate and that any inaccuracies were unbiased with respect to student condition.

Analysis: This study used three logistic regressions to estimate program effects. Control variables included demographics and the baseline outcomes of pregnancy, course failure, and academic suspension status at the beginning of the year. Checks for clustering within classroom using multilevel models were reported as a supplement.

Along with complete-case analyses, the study presented analyses using all cases under three forms of imputation. First, dropouts were assumed to have the same rate of problem behavior as completers. Second, dropouts were assumed to have twice the rate of problem behaviors as completers. Third, dropouts were assumed to have had four times and eight times the rate of problem behavior of completers (up to a 100% rate).

Intent to Treat: The study used all students with complete data in the main analysis and reported on results using all students after imputation of missing data.

Outcomes

Implementation Fidelity:

Although Teen Outreach sites were expected to provide a minimum of 20 hours per year of volunteer experience, participants averaged 37.4 hours, with a median of 25 hours. No other measures of implementation fidelity were reported.

Baseline Equivalence:

Tests reported in Table 1 showed two significant differences in 17 tests. The significant differences appeared small. Females represented 29.1% of the comparison group and 24.6% of the program participants, and 46.1% percent of the comparison group identified as black, compared with 44.3% percent of the program participants.

Differential attrition:

The Teen Outreach sample experienced an overall attrition rate of 8.9% and the comparison group exhibited a rate of 7.8%. Attriters were more likely than non-attriters to become pregnant or cause a pregnancy and/or fail courses in school. They were also more likely to be younger, male, members of a minority group, and living in single-parent households. However, the authors stated that these differences were similar across conditions (i.e., there was no interaction of attrition and condition).

Posttest:

The authors did not report results separately for the randomized sample. However, they stated that, in allowing a variable for random assignment to interact with the effects of interest, the analysis found no statistically significant differences in program effects between the randomized and non-randomized samples.

Teen pregnancy: Self-reports showed that intervention participants had significantly fewer pregnancies than the comparison group (OR = .53). In seven interaction tests, only one emerged as significant: the intervention had a stronger effect among those who were teen parents at baseline than others.

Course failure: Self-reports showed that intervention participants had significantly fewer course failures than the comparison group (OR = .60). In seven interaction tests, three emerged as significant. The intervention had stronger benefits for females than males, for racial minorities than majorities, and for students with prior suspensions than students with no prior suspensions.

Academic suspension: Self-reports showed that intervention participants had significantly fewer suspensions than the comparison group (OR = .52). No significant interaction effects emerged in seven tests.

Multiple imputation results: In the pregnancy model, all the main effects and interactions remained significant. In the course failure model, participation and interactions remained significant when imputing problem behaviors at twice the baseline rate, but not when imputing problem behaviors at four times the baseline rate.

Multilevel results. Models with random effects for schools did little to change the results.

Long-Term:

Not examined.

Study 2

Summary

Allen et al. (1997) conducted a randomized controlled trial with 695 adolescents across 25 sites nationwide. The randomization assigned students individually and, less frequently, by classrooms to an intervention group or control group. Self-report assessments of pregnancy, course failure, and academic suspension were conducted at baseline and 9 months later (posttest).

Allen et al. (1997) found, compared to the control group, participants in the intervention group experienced significantly lower risks of:

• Pregnancy
• Course failure
• Academic suspension

Evaluation Methodology

Design:

Recruitment: From 1991 through 1995, 25 schools with Teen Outreach programs participated in this evaluation. With all known Teen Outreach programs in the country contacted, 10% (n = 25) agreed to participate. Although non-random, the sample was national in scope. Students at the sites indicated interest in the program by enrolling as part of the health curriculum, as an academic elective, or as an after-school activity. Total participation included 695 students in grades 9 through 12.

Assignment: The students were randomly assigned within schools either individually when there was program over-enrollment or, less frequently, by classroom. The intervention group (n = 342) took courses that included the program curriculum and activities, while the control group (n = 353) took the usual health or social studies courses.

Assessments/Attrition: The baseline assessment came within the first few weeks of the academic year, while the posttest came at the end of the academic year. At the early stages of data analysis, two sites had attrition rates between 80 and 90 percent and one site showed a significantly higher rate of baseline problem behaviors for the control group than the program group. These three sites were removed, and 22 sites were used in the main analysis. For the remaining 22 sites, attrition was 5.3% for the Teen Outreach program and 8.4% for the control group.

Sample characteristics: The typical student in the sample was about 16 years old, in 9th or 10th grade, female (roughly 85%), and black (about 67%). The high rate of female participation is explained by teachers and counselor's viewing the program as part of pregnancy prevention and therefore recommended more females than males.

Measures: The measures came from a self-report questionnaire that program youth completed in their Teen Outreach classroom and control youth completed in study hall or other school setting. The problem behavior questions asked students whether they had 1) been pregnant or caused a pregnancy, 2) failed any courses during the prior year, and 3) been suspended academically in the prior year. In the posttest, the problem behaviors referred to the past academic year.

A validity check used data from one of the sites that provided administrative data on course failures and school suspensions. For that site, over-reporting was common but was similar across conditions and across baseline and posttest assessments.

Analysis: Logistic regression examined the effects of the program while controlling for baseline outcomes and demographic characteristics. Although some students were randomized by classroom, the analysis did not check or adjust for clustering.

Intent to Treat: Dropping three schools may have violated the intent-to-treat criterion, but the exclusions related more to data quality than low program participation, and the authors stated that including the three schools did not change the significance of the program effects.

Outcomes

Implementation Fidelity: While required to provide 20 hours of volunteer service, program participants averaged 45.8 hours. The study reported no other measures of implementation fidelity.

Baseline Equivalence:

No statistically significant differences in demographic characteristics existed across the groups (Table 1). However, the control group exhibited significantly higher rates on all three problem behavior outcomes. The differences came in part from the three sites that were dropped from most of the analysis. Among the other 22 sites, the differences may have come from completing the baseline measures a few weeks after the program start. The potential effects of baseline differences for the remaining 22 sites were examined as part of the outcome analysis.

Differential Attrition:

Among sites, no analysis was made to determine whether the three sites that dropped out of the study had students who were significantly different than the other sites demographically or in terms of problem behaviors at entry. However, an alternative analysis was conducted with the three sites included and all effects of the program remained at the same significance levels. The method used to impute data in order to run the alternative analysis was not reported.

Among students in the remaining 22 programs, dropouts were significantly different at baseline than completers on prior pregnancy, academic suspension, age, and gender. However, comparisons across conditions found that attrition was not linked to being in the control or program group (i.e., there was no interaction of attrition with condition).

Posttest:

The Teen Outreach program had statistically significant and moderate effects on all three problem behavior outcomes. The odds ratios equaled .42 for pregnancy, .39 for course failure, and .41 for school suspension. Interaction tests showed one group difference in program effects, with females benefitting more from the program than males. Further tests including the three sites dropped from the main analysis found that the program effects remained at the same significance levels.

To check for bias from condition differences at baseline among the 22 analysis sites, the study first created a dichotomous variable that indicated whether the site had a higher number of problem behaviors at entry in the control group than the intervention group. It next estimated models that allowed the program to interact with the new variable. None of these interactions was significant, leading the authors to conclude that baseline non-equivalence did not explain the observed program effects. Lastly, they dropped the four schools with the greatest imbalance, finding that the significance levels of the program effects were unchanged.

Long-Term:

Not examined.

Study 3

Summary

Allen et al. (1990) conducted a quasi-experimental study with 1,487 youth in grades 7-12. A self-report questionnaire assessing pregnancy, course failure, and academic suspension was administered within the first two weeks of the beginning of school in the fall and at program exit near the end of the academic year.

Allen et al. (1990) found, compared to the control group at posttest, program participants had significantly fewer:

• Problem behaviors

Evaluation Methodology

Design: This evaluation was designed to assess both individual and program-level factors that may influence the effectiveness of the Teen Outreach program in reducing problem behaviors in youth. Data were collected at the site- and the individual-level during the 1986-1987 academic/program year. Of the 70 Teen Outreach sites in the nation, the study sampled 35 sites in 30 schools. The authors did not indicate how the evaluation sites were selected, but reported that the sample was geographically disperse and the sampled sites did not differ significantly from the non-sampled site. The authors did not indicate which site characteristics were used to draw this conclusion.

Program participants could have entered the program by electing to participate in response to school-wide announcements or by agreeing to participate after they had been targeted by teachers or counselors. After the program youth were chosen, the comparison youth were quickly selected through one of two methods: (a) the program participants nominated youth whom they thought would fill out the questionnaire the same way they did; or (b) teachers and counselors who had made referrals to Teen Outreach identified other youth whom they thought had similar sociodemographic backgrounds and experienced similar problems as the youth chosen for the program.

Participants in the study included 632 Teen Outreach participants and 855 comparison students in grades 7-12. Attrition over the course of the program year was 2.4% among Teen Outreach students and 4.8% among comparison students. An additional 6.5% of the Teen Outreach students did not complete exit questionnaires and were excluded from the analysis. Consistent with the intent-to-treat principle, the analysis used all subjects with available data.

Sample characteristics: Students ranged in age from 11 to 19. The typical member of the sample was 16 years old and in 10th grade. Almost 70% of the sample was female, about half were white, and about a third were black. The mean parents' education for the sample was slightly more than a high-school graduate and slightly more than half of the youth lived in two-parent households.

Measures: A simple self-report questionnaire was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June).

In the entry questionnaire, students were asked a set of demographic questions (age, grade in school, race/ethnicity, mother's and father's education levels, and whether they lived in a two-parent household). Next, students were asked a set of questions about three specific problem behaviors (whether they had been suspended from school in the previous school year, whether they had failed courses in the previous school year, and whether they had ever caused a pregnancy or been pregnant).

In the exit questionnaire, they were asked the same questions regarding problem behaviors, but the time reference period was the academic year just ending. In addition to the three primary problem behaviors, they were also asked if they intended not to return to school in the fall.

The researchers chose to sum these behaviors into an entry problem behavior syndrome score (ranging from 0 to 3 problem behaviors) and an exit problem behavior syndrome score (ranging from 0 to 4 problem behaviors). By doing so, they addressed the difficulty of having low base rates on individual problem behaviors and also were consistent with the understanding at the time that a "syndrome" was more meaningful than individual inter-correlated behaviors. The mean entry problem behavior syndrome scores were .80 for the Teen Outreach group and .66 for the comparison group (p<.05).

Site-level data were collected from facilitators at each site across the following dimensions:

• Intensity measures: the average number of volunteer hours worked and the average number of classroom hours;
• Structural measures: whether the program was offered during school or after school and whether it was for-credit or not-for-credit; and
• Curriculum measures: a four point likert-scale measured how much (none, a little, a lot, almost all) facilitators used the curriculum overall and by curriculum category. The categories were identified by factor analysis to be an orientation section, a self-awareness section, a social-awareness section, and a developmental section.

Analysis: The data used for this study were multi-level in nature, but the analysis was not. Individual students were nested within programs and each was assigned the average program characteristics from that site. This evaluation conducted two primary analyses. First, to determine the general effectiveness of the program, problem behavior outcomes were compared across the Teen Outreach youth and the comparison youth. Second, among Teen Outreach programs, researchers sought to identify the role of site-level factors in the efficacy of the program. As such, the regression models were run with individual-level characteristics as controls, with site-level intensity measures as explanatory factors, with site-level structural measures as explanatory factors, and with site-level curriculum used as explanatory factors.

Outcomes

Baseline Equivalence: Compared to the Teen Outreach participants, the comparison group youth had fathers with a slightly higher level of education, were more likely to live in a two-parent household, were less likely to have failed courses in the prior year, and had fewer problem behaviors on average.

Differential Attrition: Without reporting figures, the authors report "no significant effects of loss-to-follow-up on demographic or problem behavior measures, nor were there significant interactions of loss-to-follow-up with membership in the program versus comparison group for any of these measures."

Posttest: At the beginning of the program, the Teen Outreach participants had more problem behaviors, on average, than the comparison group (p<.001). At program exit, the Teen Outreach participants had fewer problem behaviors, on average, than the comparison group (p<.05). Thus, compared to non-participants, participating in Teen Outreach was associated with a reduced number of problem behaviors.

The average number of volunteer hours at a site had a significant and positive but weak effect on reducing the number of problem behaviors (beta=-.09, p<.05). The number of classroom hours also had a positive effect in reducing the number of problem behaviors, but the effect was stronger for the younger participants (beta for grade x class hours = .10, p<.05). Whether a program was offered during the school day for credit or after-school as an activity had no significant impact in the effectiveness of the program in reducing problem behaviors. Also, the effectiveness of the program was not significantly altered by the amount of the overall curriculum used or by different usage of the categories of the curriculum.

Limitations

The major limitations of this study include:

• The evaluation's overall lack of generalizability. This inability to generalize is due to student self-selection into the program, lack of clarity around site selection, and demographic characteristics of the sample that do not generalize to the population.
• The estimates of program-level factors are made using the individual as the unit of analysis. Thus, standard errors are likely under-estimated and the results are likely biased upwards.
• Non-equivalence exists between the treatment group and comparison group, possibly because neither group was randomly selected.
• Control group participants were nominated by program participants or teachers rather than randomly selected.
• Using a problem behavior syndrome approach assumes that each of the behaviors is weighted the same. Further, the impact of the program cannot be differentiated between reducing pregnancy or academic failures.

Study 4

Summary

Allen et al. (1994) conducted a quasi-experimental study with 2,033 youth (grades 7-12) across 66 sites. A self-report questionnaire assessing pregnancy, course failure, and academic suspension was administered within the first two weeks of the beginning of school in the fall and at program exit near the end of the academic year.

Allen et al. (1994) found, compared to the control group at posttest, program participants had significantly fewer:

• Problem behaviors

Evaluation Methodology

Design: This quasi-experimental study used individual and site-level data from 66 different Teen Outreach programs across the country. The researchers did not indicate how the 66 sites were selected or from which program year(s) the data were drawn.

Participants enrolled in Teen Outreach by a variety of ways: it could have been part of the required curriculum, they could have chosen it as an elective or taken it as an after-school activity; or they could have agreed to take the program in response to a recommendation from a teacher or guidance counselor. Previous research has shown that the method of entry into the program is unrelated to outcomes. After choosing the program youth, the investigators quickly selected comparison youth through one of three methods:

1. Program participants nominated youth who they thought would fill out the questionnaire the same way they did; or
2. Program participants nominated youth whom they thought were from similar backgrounds and encountered similar life problems; or
3. Comparison youth were randomly selected from a pool of youth who had wanted to enroll in the Teen Outreach but the program was already at capacity.

Within the 66 sites, data were collected from 1,020 Teen Outreach youth and 1,013 comparison group youth in grades 7 - 12. Attrition was 2.4% among Teen Outreach students and 2.7% among comparison students.

Sample characteristics: Students ranged in age from 11 to 19 years old. The typical member of the sample was 16 years old and in the 10th grade. About 70% of the sampled students were female, and roughly 40% were black and 40% were white. The mean parents' education level was just above a high school graduate and slightly more than half of the youth came from two-parent households.

Measures: A simple self-report questionnaire was administered within the first two weeks of the beginning of school in the fall (usually August or September) and at program exit near the end of the academic year (typically May or June). Students were asked their age, their grade in school, their race/ethnicity, their parents' education level (1=not a high school graduate through 4=college graduate), whether they lived in a two-parent household, whether they had been academically suspended from school in the previous school year, whether they had failed courses in the previous school year, and whether they had ever caused a pregnancy or been pregnant. In the exit questionnaire, the reference time period was the academic year just ending.

To measure the degree to which programs fostered senses of autonomy and relatedness, a questionnaire was administered to students and to facilitators during the final month of the program. This questionnaire was intended to measure (a) the extent to which the program was viewed as promoting adolescent autonomy and relatedness; and (b) whether the quality of the volunteer experience was likely to enhance the students' sense of competence and autonomy.

The questionnaire was structured similar to the Perceived Competence Scale for Children, which is intended to reduce the effects of a pull for social desirability. For each item, two contrasting stems were presented side by side (e.g., "some kids feel like their facilitators like them a lot" and "other kids feel like their facilitator just likes them OK."). The youth were asked to pick one of the two statements that they most agreed with and then decide whether it was "really true" or "sort of true." The autonomy and relatedness questions focused on whether the youth felt like they could make their own decisions within the program, whether their views were respected, etc. The quality of volunteer experience questions focused on whether the youth felt like they were learning new skills, whether they might like to continue volunteering after the program ended, etc. The two scales had adequate internal consistency (alpha=.87 and alpha = .72, respectively).

The questionnaire was also given to facilitators who were asked to answer as they thought their class would answer. The adolescent autonomy and relatedness scale had an alpha of .81 and the quality of volunteer experience scale had an alpha of .52. The authors report that the "validity of this inventory approach has been established in other studies, in spite of its reduction of alpha levels".

Students were given the standard problem behavior questionnaire at the beginning of the program and at the end. Both program youth and facilitators were given the autonomy and relatedness questionnaire and the quality of volunteer experience questionnaire at some point in the last month of the program. Students were assured of confidentiality and that their data would not be shared with school officials.

Analysis: The data used for this study were multi-level in nature. Individual students were nested within programs and each was assigned the average program values for program level measures. This evaluation conducted two primary analyses. First, to determine the general effectiveness of the program, problem behavior outcomes were compared across the Teen Outreach youth and the comparison youth. Second, among Teen Outreach programs, researchers sought to identify whether feelings of autonomy and connectedness and quality volunteer experiences mediated the program's effectiveness. Also, to control for potential unmeasured site-wide factors, the authors included residualized problem behavior change scores for the comparison group for each site.

Linear regression models were run with the number of problem behaviors at exit as the dependent variable. The covariates were entered sequentially, as follows: individual-level data (number of problem behaviors at entry, grade level, residualized problem behavior change score); then the primary variable of interest (promotion of autonomy and relatedness scale in regression #1 and quality of volunteer experience in regression #2); and, finally, interactions of grade level with the primary variable of interest. Each model was run twice, once using student ratings and once using facilitator ratings.

Outcomes

Baseline Equivalency: The Teen Outreach participants and the comparison group were very similar with respect to demographic characteristics and program behaviors at entry. The only statistically significant difference was that the proportion of females was slightly higher among the Teen Outreach group. This difference can be attributed to adult referrals into a program that they view as primarily a pregnancy prevention program.

Differential Attrition: Attrition was similarly low in both the treatment and comparison groups.

Posttest: At program entry, there was no significant difference in the mean number of problem behaviors (ranging from 0 to 3) between the Teen Outreach youth and the comparison youth (.61 vs.64, respectively). After controlling for problem behaviors at program entry, the mean number of problem behaviors was significantly lower (p<.001) for youth in the Teen Outreach program (.48 vs.67, respectively).

When testing for site-level effects, the researchers used averaged site-level data from both the students and facilitators. In the student rating model, programs with higher scores on the promotion of autonomy and relatedness scales were significantly more effective in reducing problem behaviors (p<.01). However, this effect was only true for middle school students (p<.05). Likewise, in the facilitator model, the only significant effect of promotion of autonomy and relatedness was for middle school students (p<.01).

The regressions were run again with the quality of volunteer experience as the key explanatory variable. In the student rating model, programs with higher scores on the quality of volunteer experience scale were significantly more effective in reducing problem behaviors ( p<.05), but once again this effect only held for middle school students. In the facilitator rating model, no significant effects of the quality of volunteer experience was found. Additionally, the effect of the mean number of volunteer hours on program success was not significant.

The authors conducted an analysis to determine if the small effect found for quality of volunteer experience (i.e., enhancing autonomy and relatedness) on reducing problem behaviors for middle school students was a mediating effect of the volunteer experience itself. The authors conclude that, in fact, a portion of the volunteer experience effect is mediated through the perceived autonomy and relatedness aspects of the program.

In sum, the proposed psycho-social mechanisms were significant only for middle schoolers. This was surprising because previous research has shown that the overall Teen Outreach program is more effective, in general, for high school youth. The authors concluded that middle school youth are more in need of an environment that fosters this sort of development, while high school students may already have achieved a certain level of autonomy and relatedness. High school youth are therefore less dependent on getting this specific developmental competency within the program and instead benefit from other aspects of the program.

Limitations

The major limitations of this study include:

• The evaluation's overall lack of generalizability. This inability to generalize is due to student self-selection into the program, lack of clarity around site selection, and demographic characteristics of the sample that do not generalize to the population.
• The estimates of program-level factors are made using the individual as the unit of analysis. Thus, standard errors are likely under-estimated and the results are likely biased upwards.
• Non-equivalence exists between the treatment group and comparison group because neither group was randomly selected.
• Control group participants were nominated by program participants or teachers rather than randomly selected.
• There is no independent measure of autonomy and relatedness. The measure used in this study substantially overlaps with the students' negative or positive experience of the program.
• Success in the Teen Outreach program may influence the measure of autonomy and relatedness rather than vice-versa.

Study 5

Summary

Walsh-Buhi et al. (2016) tested the program using a cluster randomized trial of 28 public schools in 12 non-metropolitan Florida counties. A total of 7,667 students across two cohorts participated in the program. Primary outcomes regarding sexual activity included ever having had sex, recent sex, recent risky sex, and intention to have risky sex, with assessments conducted at baseline and at posttest, approximately 9 months after baseline.

Walsh-Buhi et al. (2016) found at follow-up, compared to the control group, participants in the intervention group showed no significant improvements in risky sexual behaviors overall. Cohort 1 program youth were significantly less likely than control group youth to report recent sex, risky sex, and sexual intentions, and Cohort 2 program youth had significantly reduced risky sexual intentions than control youth.

Evaluation Methodology

Design:

Recruitment: Schools were recruited for participation if they were located in nonmetropolitan Florida counties (those with a population of less than 900,000), if they had the capacity to implement the program, and if they had higher rates of adolescent births, STIs, high school dropout, graduation, or out-of-school suspension rates compared to other communities. A total of 28 public schools within 12 counties were selected for program implementation. Two cohorts of 9th grade classes were recruited for participation in the Walsh-Buhi et al. (2016) study, with 9,220 students found eligible and 7,667 consenting to participate. Daley et al. (2019) offered a longitudinal analysis of youth enrolled in the study beginning in fall 2012 (i.e., cohort 1).

Assignment: The schools were paired on structural and demographic characteristics and randomized into the intervention or control groups. Specifically, the 28 schools reported in Walsh-Buhi et al. (2016) were matched into pairs based on county, courses offered, school size, region/proximity, and presence of block scheduling, and then randomly assigned to either the intervention (n=14 schools, 3,533 students) or control group (n=14, 4,134). In the intervention group, the intervention was delivered as a supplement to the usual health curriculum while control schools continued with health education as usual. On page 16, Daley et al. (2015), reporting only on cohort 1, stated, "Passive parental consent was obtained after random assignment for a total of 4,063 youth throughout the study (1,845 intervention and 2,218 comparison)."

Attrition: One school dropped out during the first year so the school and its matched pair were dropped for cohort 2. As reported in Walsh-Buhi et al. (2016), as stated in the text on p. 285, there were 3,988 students in cohort 1 that completed a baseline measure and 3,404 at follow-up, resulting in a retention rate of 100% at the school level and 85% at the student level. For cohort 2, 3,679 students were assessed at baseline and 2,758 completed a follow-up survey for a retention rate of 93% at the school level and 75% at the student level. Overall (with both cohorts combined), the retention rate was 80% (7,667 at baseline and 6,162 at follow-up). Daley et al. (2019) reported on only cohort 1 and showed a student-level retention rate at the one-year follow-up of 67% (n = 2,660).  It is worth noting that the baseline participant numbers are reported somewhat inconsistently in Daley et al. (2019). Figure 1 reports 3,988 students with baseline data while Table 2 reports 3,836.

Sample: Across both cohorts the sample was evenly split on gender with a mean age of 14.6 at baseline. Nearly 60% of respondents were white, with 20% reporting as Hispanic, 10% as Black, and the remaining 10% as other races/ethnicities. Approximately 15% of the sample reported recent sexual activity; of those who reported recent sexual activity, 27% reported recent risky sex. About 32% of the entire sample intended to have sex within the next year, with 8% of those intending risky sex in that same time frame.

Measures: Outcomes of interest were self-reports of: ever had sex (ever sex), sexual intercourse in the previous 3 months (recent sex), not using a condom in the previous 3 months (recent risky sex), and ever been pregnant or gotten someone pregnant (ever pregnant). Secondary outcomes included intention to have sex in the next year (sex intentions), and, among those who intended to have sex in the next year, their intention to use a condom or not (risky sex intentions). Daley et al. (2015) reported on just two of these measures: ever sex and ever pregnant.

Analysis: Walsh-Buhi et al. (2016) used two-level generalized linear mixed effects models with logit link function, controlling for baseline demographics (age, gender, race/ethnicity) and pretest behaviors. School IDs were treated as the source of random effects. Daley et al. (2015) reported the use of the Bonferroni multiple comparisons correction to the ever sex question (for cohort 1). For Daley et al. (2019), three levels (i.e. students within classrooms, nested within schools in the same county) of random effects were included to account for clustering effects. The researchers also attempted to fit a model with school as a random factor, but those models had convergence problems. Repeated measures were calculated across the three time points (baseline, first follow-up, and second follow-up) for each individual, and the random variation of each observation was taken into account. All models included fixed effects for demographic variables (gender, age group, and race/ethnicity) and the treatment condition. Models also included randomization block (e.g., school pair) as a fixed effect except for the outcomes corresponding to the question "Have you ever been pregnant or gotten someone pregnant?" That outcome had low prevalence, overall, so for those models, blocks were collapsed into larger more inclusive categories based on the number of students and geographic setting of each school. Walsh-Buhi et al. (2016) ran many different subgroup analyses, including differential impacts of the program across cohorts, genders, and by baseline behaviors. Daley et al. (2019) performed a subgroup analysis by gender.

Intent-to-Treat: Multiple imputation was used to account for missing data, with all cases used in the analyses for both Walsh-Buhi et al. (2016) and Daley et al. (2019).

Outcomes

Implementation Fidelity: Fidelity was not assessed in Walsh-Buhi et al. (2016). As reported in Daley et al. (2015, 2019), for cohort 1, the intervention was implemented in 70 individual classes. Of the 70 classes, 51 classes (73%) offered at least 25 weekly sessions, as prescribed (range: 23-57 sessions). Relatively few classes-8 (or 11%)-received at least 20 community service project hours, as prescribed. No classes received programming over a span of 9 months.

Baseline Equivalence: As reported by Walsh-Buhi et al. (2016) in Tables 2 and 3 (p. 287), there were no significant differences between groups at baseline for the combined cohorts. When examined by cohort, out of 3 socio-demographic and 10 pretest variables tested, there was one baseline difference for cohort 1 (race/ethnicity). For Daley et al. (2019), out of the 3 socio-demographic (Table 2, p. 8) and 6 pre-test measures (Table 3, p. 9), no significant baseline differences were detected. As for the analysis sample, Walsh-Buhi et al. (2016) and Daley et al. (2019) state that they tested for condition equivalence of those retained at posttest; neither presented the results of the attrition analyses. Daley et al. (2015), however, reported no demographic or pretest differences between conditions for the analysis sample with one exception, youths in the control group were more likely to answer "yes" to the ever had sex question at both the posttest and follow-up (Tables III3 and III4).

Differential Attrition: As mentioned previously, equivalence for the analysis sample was not presented in either Walsh-Buhi et al. (2016) or Daley et al. (2019), but some differences (in favor of the treatment group) were reported in Daley et al. (2015). Daley et al. (2015) reported that at both the posttest and follow-up, completers (i.e., youth with complete follow-up information who also answered each question at baseline) were significantly more likely to be younger, female, and more likely to report being White or 'other' rather than Black compared to attritors (i.e., those missing at the follow-up). Completers were also less likely to answer 'yes' to both the "ever had sex" pretest question and the "ever been pregnant or got someone pregnant" question at baseline. Differences between completers and attritors were not reported in Walsh-Buhi et al. (2016) or Daley et al. (2019).

Posttest: Overall, at posttest, there were no significant differences reported in Walsh-Buhi et al. (2016) between the treatment and control groups on any measures using the pooled cohort 1 and cohort 2 data. However, impacts were much more positive in cohort 1. Walsh-Buhi et al. (2016) found that treatment participants in the first cohort were significantly less likely than control group youth to report recent sex (OR = 0.70), risky sex (OR = 0.71), and sexual intentions (OR = 0.81). Daley et al. (2015) found for cohort 1, compared to control, exposure to treatment significantly reduced the number of youth ever having sex by 3.7% and significantly reduced the rates of ever being pregnant or causing a pregnancy by 1.14%. Reporting on subgroup effects, Walsh-Buhi et al. (2016) found that females showed lower odds of recent and risky sex, while males showed decreased odds of recent sex. Of those inexperienced at baseline, intervention group participants were significantly less likely to report both ever having had sex and recent sex; females were significantly less likely to report ever having been pregnant, ever having sex, and having recent sex. Of those in cohort 1 considered experienced at baseline, intervention participants were significantly less likely to report risky sex; females were significantly less likely to report recent sex and risky sex. There was an iatrogenic effect of the program for cohort 1 males who were sexually inexperienced at baseline, but sexually active at follow-up, as they actually displayed greater odds of risky sex than their control group counterparts.

For Walsh-Buhi et al. (2016), the intervention participants in cohort 2 reported significantly reduced intentions to have risky sex compared to the control group. However, of those considered experienced at baseline, intervention participants had significantly higher odds of recent sex than those in the control group - an iatrogenic effect.

Long-Term: Daley et al. (2019) found that, for cohort 1, compared to the control group, ten to 12 months after the intervention ended, the treatment group was less likely to report ever having been pregnant or having got someone pregnant (OR = 0.55). There was no long-term impact, however, on having sex (Daley et al., 2015, 2019). When examining treatment effects by gender, statistically significant differences were found only for female participants. No significant difference between treatment and control groups were found at the follow-up for the outcomes sex in the past 3 months, sex without a condom in the past 3 months, intention to have sex, and intention to not use a condom. Researchers did find an effect of the treatment on ever having sex among males at second follow-up (Daley et al., 2019), but not at first follow-up (Walsh-Buhi et al., 2016).

Study 6

Summary

Robinson et al. (2016) evaluated the program in a community setting. The researchers randomly assigned 4,769 youth from Louisiana and 966 youth from Rochester, NY to the intervention and control conditions. Measures of sexual initiation and use of birth control were collected through self-reports at baseline and immediately after the intervention period.

Robinson et al. (2016) reported no significant effects on initiation of sexual intercourse or use of birth control.

Evaluation Methodology

Design:

Recruitment: The researchers recruited 4769 youth in Louisiana and 966 youth in Rochester via convenience sampling methods. Recruitment took place in school and community settings during the academic years of 2013-2015, and the sample contained three "annual cohorts." Eligible participants included those who lived in the targets areas, spoke English well enough to complete the study materials, and had not previously participated. In Louisiana youth aged 12 to 17 years were included and in Rochester youth aged 11 to 14 years were included.

Assignment: Individuals were randomly assigned to the conditions in Louisiana and 33 clusters were randomly assigned to the conditions in Rochester. In Louisiana the control condition received no programming, and in Rochester the control condition received a work-readiness intervention.

Attrition: Of the 4769 youth in Louisiana, 2428 (51%) completed the follow-up survey, and of the 966 youth in Rochester, 824 (85%) completed the follow-up survey.

Sample:

The sample targeted areas with high rates of adolescent pregnancy and included multiple regions across Louisiana and Rochester. In Louisiana, the sample was 90% black and 60% female, with an average age of 13.9 years (SD = 1.59). In Rochester, the sample was 64% black and 52% female with a mean age of 12.3 years (SD = 1.11).

Measures:

Participants completed self-report outcome measures. Sexual activity was measured through a yes/no question about whether the participant had previously engaged in intercourse; the posttest initiation measure included only those who had not already had intercourse at baseline. A yes/no question asked about whether the participant had engaged in intercourse without birth control in the previous 3 months.

Analysis:

Logistic regression models were used to analyze predictors of the binary outcomes, with gender, age, race, ethnicity, and cohort year used as covariates. The baseline outcome was included in models of the use of birth controls but not for initiation of sexual intercourse (all analyzed subjects had not previously engaged in sexual intercourse). The analysis was done at the individual level with multivariable logistic regression in Louisiana, and at the clustered level with multilevel modeling in Rochester.

Intent-to-Treat: The researchers stated that they used intent-to-treat analyses by including all individuals who completed the posttest. However, they also stated that "During the course of the study, 3 study sites were dropped from the program."

Outcomes

Implementation Fidelity:

The researchers reported that "process measures" were recorded after each session by facilitators to assess fidelity. Approximately 10% of sessions were observed and evaluated as well and high ratings were given by evaluators. However, only 21% of youth in Louisiana and 48% of youth in Rochester in the intervention condition attended at least 75% of the sessions.

Baseline Equivalence:

The researchers tested for baseline equivalence between the intervention and control conditions on demographic variables and outcome measures and found no significant differences in Louisiana. In Rochester, the intervention group included significantly more Hispanics. However, the tests used the analysis sample of those with posttest data rather than the randomized sample.

Differential Attrition:

Differential attrition was not analyzed directly. Tests for baseline equivalence for the analysis sample are relevant but lacked comparison to the randomized sample, included only 5-6 baseline measures, and showed one difference.

Posttest:

No significant effects on outcome variables were found.

Long-Term:

Long-term effects were not tested.

Study 7

This study compared the TOP program with a text message supplement to the standard TOP program.

Summary

Bull et al. (2016) conducted a cluster randomized control trial between September 2011 and September 2014 with 852 youth across 32 Boys & Girls clubs in Denver, CO. The control condition consisted of TOP programming alone, and the intervention condition consisted of TOP programming plus the text message supplement. Participants completed self-report measures at baseline (pretest) and immediately at program completion, after four years of programming (posttest).

Bull et al. (2016) found no effects overall but in comparison to the control condition, the intervention condition showed significantly fewer pregnancies for Hispanic participants only.

Evaluation Methodology

Design:

Recruitment: Participants were recruited from eight Boys & Girls clubs in Denver over 4 years. Eligible participants included active club members between 14 and 18 years of age. A total of 3643 youth fit all eligibility criteria, and of these, 852 consented to participate in the study.

Assignment: Randomization was clustered by site and by year, creating 32 clusters in total. Each site was assigned to the intervention condition for two of the years and to the control condition for the other two years. The intervention sites received TOP programming plus the text-message-based supplement, and control sites received just the TOP programming. The sample included 436 in the intervention condition and 315 in the control condition at baseline.

Attrition: Participants completed self-report measures at baseline and immediately at program completion. At posttest, 72.7% of the intervention group and 75.7% of the control group completed the assessment.

Sample:

The clubs primarily served minority youth and those living in poverty. The mean age was 14.90 (SD = 1.02) in the intervention condition and 14.98 (SD = 1.14) in the control condition. The intervention condition was 42.41% Hispanic, 20.25% black, and 9.18% white, and the control condition was 43.81% Hispanic, 18.73% black, and 7.94% white.

Measures:

Measures were administered through self-reports. Participants reported how many times they had had sex in the previous three months and how many of these times were without protection. They also reported whether they had "accessed contraceptive or sexually transmitted infection services in the past 9 months" and whether they had gotten pregnant. The study reported no information on the validity and reliability of the measures.

Analysis:

Multilevel regression models were used to analyze the outcome variables, with a large number of covariates including the baseline outcomes. Sub-analyses compared Hispanic to non-Hispanic participants. Bonferroni-Hochner corrections were used to correct for multiple tests.

Intent-to-Treat: The authors stated that subjects were used in the originally assigned conditions, regardless of the level of engagement with the program. Also, sensitivity analysis included missing data with full information maximum likelihood estimation and gave the same results.

Outcomes

Implementation Fidelity:

No information on implementation fidelity was given, but the subjects attended an average of 11.9 sessions of the 25.

Baseline Equivalence:

Table 2 showed only 1 significant difference out of 22 tests, but the table used the analysis sample. The significant difference was for pregnancy.

Differential Attrition:

Differential attrition was tested by condition and by 22 baseline variables. Those remaining in the study reported higher rates of protected sex and living with both parents and lower likelihood of having sex. Additionally, intervention participants lost to follow-up were significantly older than those who remained in the study. However, tests for the interaction of the baseline variables and condition showed that the differential attrition was similar across conditions. The tests for baseline equivalence using the analysis sample also suggest a limited influence of attrition.

Posttest:

The program showed no significant effects for the full sample. Hispanic participants in the intervention condition reported significantly fewer pregnancies than those in the control condition, but the authors noted that this effect may be influenced by the significant baseline difference between conditions for pregnancy.

Long-Term:

No long-term effects were reported.