Communities That Care

Study 1

Summary

The Community Youth Development Study [CYDS] consists of 20 articles (three certified by Blueprints: Hawkins, Brown et al., 2008; Hawkins et al., 2009, 2012) that examined a cluster randomized controlled trial of 24 communities across seven states. After matching, 12 communities were assigned to the intervention group and 12 to the control group. A total of 4,407 fifth-grade students in the 24 communities were surveyed nine times from 2004 to 2014, roughly from age 12 to age 23. Outcomes included initiation and prevalence of substance use and delinquent behavior.

Through Grade 7 (Hawkins, Brown et al., 2008):

• Students in control communities were significantly more likely to initiate delinquent behavior between fifth and seventh grades than were students in CTC communities.

Through Grade 8 (Hawkins et al., 2009):

• The incidence of delinquent behavior, alcohol, cigarette, and smokeless tobacco initiation were significantly lower in CTC than in control communities between grades 5 and 8.
• In grade 8, the prevalence of alcohol and smokeless tobacco use in the last 30 days, binge drinking in the past 2 weeks, and the number of different delinquent behaviors committed in the past year in grade 8 were significantly lower in CTC communities compared to control communities.

Through Grade 10, one year after the end of technical assistance (Hawkins et al., 2012):

• The incidence of alcohol use, cigarette use, and delinquency was lower by grade 10 among students in CTC communities than in control communities.
• The prevalence of current cigarette use and past-year delinquent and violent behavior were significantly lower in CTC than in control communities in grade 10.

Through Grade 12, eight years post-baseline (Hawkins et al., 2014; Rowhani-Rahbar et al., 2023):

• Abstinence from drug use, drinking alcohol, smoking cigarettes, and engaging in delinquency were lower in the CTC communities than control communities.
• Students were less likely to ever have committed a violent act or to have carried a handgun in CTC communities, relative to control communities.

Through age 19, nine years after baseline (Oesterle et al., 2015): No main effects.

Through age 21, ten years after baseline and compared to control participants (Oesterle et al., 2018), treatment participants had:

• Higher likelihood of sustained abstinence from gateway drugs (alcohol, tobacco, and marijuana) and marijuana
• Higher likelihood of abstaining from antisocial behavior
• Reduced risk of lifetime engagement in violence

Through age 23, 12 years after baseline and compared to control participants (Kuklinski et al., 2021), treatment participants had significantly lower:

• Alcohol use,
• Illicit drug use, and
• Anti-social behavior.

Community-Level Prevention Service System Outcomes (Brown et al., 2007; Rhew et al., 2011 draft):

• CTC communities exhibited significantly greater increases in adopting a science-based approach to prevention, collaboration across community sectors, and collaboration regarding specific prevention activities between 2001 and 2004, relative to control communities.
• CTC communities reported higher levels of adoption of a science-based approach to prevention in 2009, 1.5 years after study-funded resources for CTC ended.

Significant Program Effects on Risk and Protective Factors:

• The levels of risk factors targeted by CTC communities were significantly lower among panel students in grade 7 in intervention communities than in control communities after 1.67 years of implementing preventive interventions selected through the CTC process (Hawkins, Brown, et al., 2008).
• Mean levels of targeted risks increased less rapidly between grades 5 and 10, and were significantly lower in grade 10, in CTC than in control communities (Hawkins et al., 2011).
• Protective factors of opportunities for prosocial involvement, recognition for prosocial involvement, and social skills were higher in grade eight in intervention communities than control communities (Kim et al., 2015).

Evaluation Methodology

Design:

Recruitment: With the aid of drug abuse prevention agencies in seven states, the Community Youth Development Study (CYDS) identified 20 communities that were trying to implement risk- and protection-focused prevention services and 21 other matched communities that were not using a risk- and protection-focused approach. The matching was done within each state on population size, racial and ethnic diversity, economic indicators, and crime rates (one community had two matched comparison communities). Of the 20 pairs of communities, 13 were eligible because they had not yet selected or tested effective preventive interventions, and 12 of the 13 pairs agreed to participate. The final sample thus included 24 communities.

Hawkins, Brown et al. (2008) and Hawkins et al. (2009) examined students from 88 schools in the 24 communities who were in 5th grade before the start of the program (2003-2004). Hawkins et al. (2009) referred to 4,407 consented students (76.4% of those eligible). As noted in Hawkins, Brown et al. (2008), the consented number excluded 13 students who were absent during scheduled data collection dates.

Assignment: Members in each matched pair of eligible communities in a state were randomized to the intervention or control condition by a coin toss prior to recruitment. The 12 intervention communities selected 13, 16, and 14 different prevention programs to implement during the school years of 2004-2005, 2005-2006, and 2006-2007, respectively. The number of consented students was 2,405 in the intervention communities (76.2% of those eligible) and 2,002 in the control communities (76.7% of those eligible).

Assessments/Attrition: Hawkins, Brown et al. (2008) and Hawkins et al. (2009) examined four surveys. A baseline assessment of the fifth-grade students in spring 2004 (wave 1) was followed by assessments for sixth-grade students in spring 2005 (wave 2), seventh-grade students in spring 2006 (wave 3), and eighth-grade students in spring 2007 (wave 4). The last follow-up came about 2.6 years after first implementation of the selected programs but during a period of continued technical assistance from the program.

For the first four surveys, Hawkins, Brown et al. (2008) and Hawkins et al. (2009) reported that among the sample of 4,407 consented students, 26.5% did not have fifth-grade baseline data because they were recruited in sixth grade. With the new additions included, data were missing for 3.9% and 3.8% of the students at the sixth- and seventh-grade follow-ups. In addition, missing data reached 10% for some survey items, and 3.7% of the students were dropped because they did not meet the validity screen. However, both studies appear to have excluded 388 consented students (9% of the sample of 4,407) who moved out of the school district before participating in the study for one semester; students who participated for at least one semester in grade six were tracked and surveyed.

Four long-term surveys followed the initial surveys.

• Hawkins et al. (2012): wave six, spring 2009, grade 10, six years after the program start, one year after the end of program technical assistance, 92% valid survey completion among those who consented.
• Hawkins et al. (2014) and Rowhani-Rahbar et al. (2023): wave seven, spring 2011, grade 12, eight years after program start, three years after the end of program technical assistance, 91% valid survey completion among those who consented.
• Oesterle et al. (2015): wave eight, spring 2012, age 19, nine years after program start, four years after the end of program technical assistance, 89% valid survey completion among those who consented.
• Oesterle et al. (2018): wave nine, spring 2014, age 21, 11 years after program start, six years after the end of program technical assistance, 91% valid survey completion among those who consented.
• Kuklinski et al. (2021): age 23, 12 years after the program start. The CONSORT diagram included in supplementary files shows that about 88% of the randomized participants still living provided data.

Sample:

The sample communities had populations ranging from 1,500 to 50,000 residents. The enrolled student population in a single grade in these communities ranged from 40 to 485, with two communities exceeding 400 students per grade. The sample of students was split evenly between males and females. About 70% were white or Caucasian, 9% were Native American, 4% were African American, and 20% were of Hispanic origin. The students were an average of 11.1 years of age at baseline (grade 5).

Measures:

Hawkins, Brown et al. (2008) and Hawkins et al. (2009, 2012) examined three sets of self-reported outcomes. The authors provided little information on reliability and validity, but to minimize reporting bias, they used standard measures, an anonymous survey to help ensure confidentiality, and a validity screen to indicate misreporting.

• Targeted risk factor. A single measure of the targeted risk factor combined multiple items into a standardized scale. The targeted risk factor (e.g., family conflict, academic failure, friends who engage in delinquent behavior) and the specific measure varied across communities. A community's risk factor was therefore compared to its matched control community rather than to the average across all communities.
• Targeted protective factors. Kim et al. (2015) examined 15 scales measuring protective factors relating to community, family, school, and peers. The alpha values for the scales ranged from .66-.90.
• Onset of delinquent behavior. Students reported the first occurrence and recent prevalence of any one of multiple delinquent behaviors. More severe forms of delinquency were added to later waves as they became developmentally appropriate. At the age 21 follow-up, the measure referred to anti-social behavior.
• Onset of substance use. Students reported the first lifetime use and recent prevalence of multiple substances, including alcohol, marijuana, cigarettes, and other illicit drugs (e.g., inhalants, cocaine, barbiturates, ecstasy, prescription drugs).

Hawkins et al. (2014) and Oesterle et al. (2015, 2018) added a measure of sustained abstinence from substance use and delinquent behavior. Oesterle et al. (2015) added secondary outcomes that measured substance use disorders, major depression, suicidality, educational attainment, and sexual risk behaviors. Rowhani-Rahbar et al. (2023) measured participant self-reports of having carried a handgun in the past year. Kuklinski et al. (2021) used measures of substance use and antisocial behavior as primary outcomes and measures of four-year college completion, major depressive disorder, and generalized anxiety disorder as secondary outcomes.

Analysis:

Hawkins, Brown et al. (2008) and Hawkins et al. (2009) used mixed models that accounted for the hierarchical structure of the data. The specific estimates came from linear, logistic, and survival models depending on the outcome. The models controlled for student and community characteristics and baseline outcomes, except for the incidence or onset outcomes used in survival analyses. The intervention effect came from estimating the average difference within the matched pairs; the degrees of freedom for this effect equaled the number of community-matched pairs (12) minus the number of community-level covariates and intervention effect (3) minus 1 (i.e., df = 8). However, the level-2 sample size of 24 may not be large enough to accurately estimate the standard errors, and the result may be to overstate the significance of the tests.

The analyses of initiation or onset dropped students who had already initiated at baseline. Hawkins, Brown et al. (2008) dropped 22.2% for delinquent behavior and 27.5% for substance use. Hawkins et al. (2009) dropped 21.5% for alcohol use, 8.3% for cigarette use, 2.4% for smokeless tobacco use, 0.6% for marijuana use, 8.6% for inhalant use, and 21.2% for delinquent behavior. All analyses used multiple imputation for missing data resulting from non-response of eligible participants and incomplete items of responders.

Hawkins et al. (2012, 2014), Oesterle et al. (2015, 2018), Rowhani-Rahbar et al. (2023), Kuklinski et al., 2021). Depending on the outcome, the long-term analyses adjusted for clustering of time within students and students within communities with three-level growth models, multilevel discrete time survival models, and generalized linear mixed models (linear, Poisson). All models included student and community covariates and, if possible, the baseline outcome. Degrees of freedom equaled the number of matched pairs (12) minus the number of community level covariates. All the studies imputed missing follow-up data. Models of abstinence and incidence of substance use or delinquency included only those who had not initiated those behaviors at baseline.

Rhew et al. (2018). One non-ITT analysis used inverse probability weights to compare intervention and control youths who remained in their original community through eighth grade. The inverse probability weights came from propensity scores for staying in the community, as predicted by a set of baseline sociodemographic and outcome characteristics. The analysis thus examined a subsample consisting of 3,765 youths who stayed in the communities, and the inverse probability weights were used to adjust for selection differences and make the conditions similar for the subsample. The weighted estimates also adjusted for clustering and imputed missing data.

Intent-to-Treat: Hawkins, Brown et al. (2008) and Hawkins et al. (2009) noted that students in the panel who remained in the sample communities for at least one semester in grade six were tracked and surveyed, even if they left the community. In addition, the analyses used multiple imputation for missing data on student and community characteristics, drug use and delinquent behavior outcomes, and community membership. However, it appears that 388 students (9%) in the randomized communities were not followed because they were not present for one full semester of the study, an exclusion possibly related to lack of program participation and possibly a violation of intent-to-treat.

Otherwise, one non-ITT study (Rhew et al., 2018) examined a subsample of youths who remained in their original community through eighth grade, but the analysis adjusted for selection differences across conditions with inverse probability weights.

Outcomes

Implementation Fidelity:

Quinby et al. (2008). CTC implementation fidelity ratings averaged across three groups of raters show that between 89% and 100% of the CTC milestones in the first four phases of CTC implementation were "completely met" or "majority met" in the 12 intervention communities, indicating that the first four phases of the CTC system were well implemented in the communities in this trial. There was high overall agreement (95%) in milestone ratings among the different groups of raters. Five milestones were rated as either "very challenging" or "mostly challenging" for half or more of the communities: preparing archival data to supplement the CTC Youth Survey Data, identifying resources required for new programs and policy implementation, addressing readiness issues, securing a champion, and engaging all stakeholders to support the community action plan.

Fagan et al. (2008a). By using the CTC model to select and monitor the quality of prevention activities, the 12 CYDS communities replicated 13 prevention programs with high rates of adherence to the programs' core components and in accordance with dosage requirements regarding the number, length, and frequency of sessions. Adherence scores ranged from 73% to 99% across all program replications, indicating that program staff taught the majority of program objectives and ensured completion of most of the program components. Dosage scores were also high, as 94% of the dosage criteria were met across all communities. In addition, 81% of the program cycles delivered all required lessons, in the specified amount of time, and with the recommended frequency of delivery.

Fagan et al. (2012). During the active phase of research (2007, 3.5 years after CTC was begun in communities) and at the end of grant-supported activities (2010, 6.5 years after baseline and 1.5 years following the end of training, TA, and funding to intervention communities), according to agency directors and program providers, CTC communities implemented significantly more tested and effective prevention programs and had higher rates of program sustainability compared to control communities at both time periods. CTC sites reached more children and families with prevention services at each time period, but this was only significant during the intervention stage. Only one significant fidelity effect emerged, which indicated that CTC sites provided more program oversight during the sustainability phase.

Fagan et al. (2008b). According to implementer and observer reports, across all 16 programs implemented in CTC communities, large proportions of required material were taught and core components delivered, nearly all lessons were offered in accord with the length and frequency specified by program developers, implementers were prepared and enthusiastic and used a variety of teaching techniques to convey material, and high levels of engagement from program participants were observed.

Baseline Equivalence:

Hawkins, Brown et al. (2008) reported a test for one baseline outcome, finding no significant difference for the targeted risk measure. Hawkins et al. (2009) reported more detail, finding no statistically significant condition differences for drug use or delinquency (p. 793) and similar community sociodemographic characteristics across conditions (Table 1).

Hawkins, Catalano et al. (2008) used a different baseline sample to evaluate baseline equivalence of the intervention and control communities. They examined three cross-sectional surveys of students: 14,293 in 1998, 12,992 in 2000, and 14,910 in 2002. Students in three grades (sixth, eighth, and tenth) were examined separately for 11 measures of substance use and delinquent behavior (33 measures in total). The first set of tests focused on condition differences in 2002, the last survey year; the second set of tests focused on condition differences in the change from 1998 to 2002. Combined with the 33 measures, the two sets of comparison defined 66 tests. Five of these tests reached significance. First, for 2002, significantly more eighth-grade students in control communities reported having attacked someone with the intention of hurting them. Second, for the 1998-2002 period, the trends differed significantly for eighth-grade family history of substance use (a larger increase in the control communities), 10th-grade binge drinking (prevalence diminishing more in control communities), 10th-grade 30-day marijuana use (prevalence diminishing more in control communities), and being drunk or high at school (prevalence diminishing more in control communities).

Differential Attrition:

For the surveys through grades seven and eight, only about 4% of eligible students did not complete the follow-ups, and Hawkins et al. (2009) added that "There was no systematic bias from differential accretion or differential attrition in control and intervention conditions (analyses not shown)."

For the long-term surveys, attrition ranged from 8-11%. All addressed differential attrition, though they did not present the analyses. Hawkins et al. (2012) stated only that, "There was no systematic bias from differential accretion or differential attrition in control and intervention conditions." Hawkins et al. (2014) stated only that, "Accretion and attrition were similar in both intervention groups." Oesterle et al. (2015) stated only that, "Retention did not differ by intervention condition, but was higher for females than males (94 vs. 88 %, respectively)." Oesterle et al. (2018) reported that retention was significantly higher in the intervention than the control group (92% versus 90%) and among females than males (95% vs. 88%), but did not differ by race, ethnicity, or parents' education.

Posttest:

Hawkins, Brown et al. (2008). The results through grade seven showed two significant program effects in three tests. First, the targeted risk factor at grade seven was significantly higher for students in control than intervention communities (d = .15). Second, the hazard for the onset of delinquent behavior was significantly higher in the control than intervention communities (OR = 1.27). Third, the conditions did not differ significantly in the onset of substance use.

Hawkins et al. (2009). The results through grade eight showed eight significant effects in 15 tests. Students in the control communities reported significantly higher incidence of alcohol use, cigarette use, smokeless tobacco use, and delinquency than students in the intervention communities (odds ratios ranged from 1.41 to 2.34). Students in the control communities also reported higher prevalence of alcohol use, smokeless tobacco use, binge drinking, and delinquency than students in the intervention communities (odds ratios ranged from 1.25 to 1.79).

Brown et al. (2013). Higher levels of community adoption of a science-based approach to prevention in 2004 predicted significantly lower levels of youth problem behaviors in 2007. Effects of the CTC intervention on youth problem behaviors were mediated fully by community adoption of a science-based approach to prevention, as reported by key community leaders.

Kim et al. (2015). Results in grade eight (see Figure 2) showed that the youths in intervention communities scored significant better for three of the 16 protective factor scales than youths in control communities: opportunities for prosocial involvement, recognition for prosocial involvement, and social skills.

Rhew et al. (2018). Non-ITT results in grade eight based on inverse propensity weights examined youths who remained in their original community through grade eight. Of seven substance use outcome measures and one delinquency measure (Table 3), four showed significant effects of the intervention in the weighted models for stayers. The intervention group stayers reported significantly lower past month alcohol use, past two-week binge drinking, past month smokeless tobacco use, and past year delinquent behavior than control group stayers.

Long-Term:

To briefly summarize the more detailed description of the long-term results listed below, consistent intervention effects appear through grades 10 and 12 (one year and three years after the posttest). With a few isolated exceptions, however, the benefits were largely absent at ages 19 and 21 (four and six years after the posttest).

Hawkins et al. (2012) - Wave 6, Grade 10 (one year after the end of technical assistance)
In 20 tests of intervention effects, six were statistically significant. Compared to the control group, the intervention group had significantly fewer risk factors (d = -.12), significantly later initiation (of the subsample that had not yet started) for alcohol use (OR = .62) and cigarette use (OR = .54), and significantly lower prevalence of cigarette use (OR = .79), delinquent behavior (OR = .83), and use of violence (OR = .75).

Hawkins et al. (2014), Rowhani-Rahbar et al. (2023) - Wave 7, Grade 12 (three years after the end of technical assistance). In 37 tests of intervention effects (Hawkins et al., 2014), seven were statistically significant. The most consistent effects showed for sustained abstinence (among those not having started at baseline). The intervention group had significantly higher abstinence rates than the control group for any drug use. Rowhani-Rahbar et al. (2023) found that, from grade 6 through grade 12, youths in the program communities were significantly less likely to report handgun carrying in the past year (odds ratio = .76).

Oesterle et al. (2015) - Wave 8, Age 19 (four years after the end of technical assistance). In 46 tests for main effects using the full sample (see Appendix C), three reached statistical significance, but two of those effects were iatrogenic. A global test adjusting for the multiple comparisons was not significant, which discounts the individual significant effects in either direction.

Moderation tests examined intervention effects separately for males and females but revealed few differences. The 32 tests in Figures 2 and 3 show two significant effects for males (lower prevalence of cigarette use and lower delinquency) and no significant differences for females. However, tests for interactions by gender in Appendix D, which show if the intervention effects differ significantly across men and women, indicated only one significant difference in 72 tests. The difference was for an iatrogenic effect among females of ecstasy use; the effects on the prevalence of cigarette use and delinquency did not differ significantly between males and females.

Oesterle et al. (2018) - Wave 9, Age 21 (six years after the end of technical assistance). In 16 tests for main effects using the full sample (Table 1), three reached statistical significance. For the subsample of non-initiates at baseline, the intervention group reported significantly greater abstinence of gateway drugs (d = .26) and anti-social behavior (d = .14) and significantly lower incidence of violent behavior (d = -.12). However, a global test adjusting for multiple comparisons showed that "the CTC system's overall long-term impact across all primary outcomes through age 21 years did not achieve statistical significance." In addition, there were no significant effects on 17 prevalence measures (Appendix A).

Moderation tests in Table 1 found one significant gender interaction in 16 tests for incidence of use of other drugs, but the separate intervention effects on that outcome proved non-significant for both males and females (Table 2). Despite the lack of significant differences between genders, the authors reported significant intervention effects among males only for abstinence of gateway substance use, tobacco use, marijuana use, and anti-social behavior and for incidence of inhalant use.

Kuklinski et al. (2021) - Wave 10, Age 23. Table 2 shows that three of five primary outcomes and zero of three secondary outcomes were significantly affected by the program. The young adults in the intervention group reported significantly lower alcohol use, illicit drug use, and anti-social behavior than the young adults in the control group.

Community-Level Prevention Service Outcomes:

Brown et al. (2007). CTC communities exhibited significantly greater increases in adopting a science-based approach to prevention, collaboration across community sectors, and collaboration regarding specific prevention activities between 2001 and 2004 relative to control communities.

Rhew et al. (2011). CTC had evidence of sustained effects on prevention system characteristics 1.5 years after study-funded resources for CTC ended. Leaders from CTC communities reported higher levels of adoption of a science-based approach to prevention and a higher percentage of funding desired for prevention activities in 2009 than did leaders in control communities. CTC communities showed a higher increase over time in community norms against adolescent drug use compared to control communities. There was no evidence of a main effect of CTC on community collaboration for prevention, general community support for prevention, or use of the social development strategy.

Gloppen et al. (2012). The extent to which coalitions continued to meet specific benchmarks was measured 20 months following the end of study support for CTC. Only 1 of 12 CTC coalitions had dissolved by winter of 2009. Although funding had decreased in 7 of the 11 coalitions, two-thirds of the coalitions reported having a paid staff person. CTC coalitions continued to report significantly higher scores on the benchmarks of phases 2 through 5 of the CTC system than did the prevention coalitions in the control communities, indicating that CTC coalitions maintained a more scientific approach to prevention than coalitions that did not receive CTC training.

Kuklinski et al. (2013). Many CTC coalitions were resilient two years after the economic downturn that began December 2007. CTC coalitions continued to implement science-based prevention to a significantly greater degree than control coalitions (7 of 11 coalitions maintained 2007 implementation levels through 2009).

Study 2

Summary

Feinberg et al. (2007, 2010) conducted a quasi-experimental study of CTC in Pennsylvania sites using a state-funded survey of adolescents in grades 6, 8, 10, and 12 in public and private schools. The full data sets available contained data on 92 school districts and 43,842 students in 2001 and 159 school districts and 101,988 students in 2003. Outcomes included student self-reports of substance use and delinquency.

Feinberg et al. (2007):

• In general, the pattern of findings shows that communities employing the CTC model had lower levels of risk factors and problem behaviors (delinquency and alcohol/drug use) than communities not employing CTC.
• When contrasting grade cohorts that were actually exposed to evidence-based programs ("expected impact" cohorts), compared to grade cohorts in the same schools that were not exposed to EBPs combined with students from non-CTC schools, youth in "expected impact" CTC grade cohorts demonstrated significant and beneficial effects for risk/protective factors, academic grades, and delinquency.

Evaluation Methodology

Design: Data was obtained utilizing the results from a state-funded surveillance survey of a large sample of adolescents in public and private schools, the Pennsylvania Youth Survey (PAYS). Although not designed as an evaluation of CTC, PAYS utilized the CTC Youth Survey. The survey results included communities that do and do not have CTC coalitions. The PAYS data was collected in 2001 and 2003, by the PA Commission on Crime and Delinquency. In both years, the sample consisted of students in schools that participated as part of a stratified random sampling procedure or schools that volunteered to participate in the survey. In 2001 and 2003 a random sample of schools was chosen to participate as follows: Schools with over 19 (2001) or 50 (2003) students were divided into 6 regions of the state, and for each of the 4 targeted grades (6, 8, 10, and 12) in each region, a separate random sample was drawn. Each school's grade was assigned a likelihood of participation equivalent to the proportion of the regional student population comprised by the school's grade. In addition to the targeted grades, "piggyback" grades at a school could be included in the survey at a financial discount to the school over the typical survey cost. This procedure yielded 43,842 respondents in 2001 and 38,845 respondents in 2003. In 2003, additional schools volunteered to participate in the survey in order to monitor risks and problems in their communities. The full data sets available contained data on 92 school districts and 43,842 students in 2001 and 159 school districts and 101,988 students in 2003. A total of 50 school districts in 2001, and 102 districts in 2003 were associated with a CTC site. For 2001, the mean time since the initial grant began was 16.8 months. For 2003, the mean time was 42.9 months.

From the full data sets, data was removed for (1) 3,752 students in 2003 because the school district identifier was missing in the data set; (2) 6 school districts whose association with CTC could not be determined; and (3) private schools because CTC sites typically focus programs in public, not private contexts.

Sample: The school districts in the combined 2001-2003 sample had an average of 6% of households below the poverty line (range 1 to 22); and an average of 7.9% female-headed households (range 3-35.2). Although many of the school districts in the rural and small town areas were predominantly white, some areas had predominant minority populations. The average percentage for nonwhites was 6.1 (range 0.7 to 77.8%), and the average percent Hispanic was 2 (range 0.2 to 22.1%).

Measures: The CTC Youth Survey (CTCYS) was the student self-report measure used to assess risk and protective factors for adolescent behavior problems as well as substance use and delinquency. Please see the Study 1 write-up for a detailed description of the CTCYS. The 6 outcome measures used for this study included an 8-item scale assessing delinquent behaviors in the past year; a seven-item "drug involvement" scale assessing use of several substances in the past 30 days (alcohol, smoking and smokeless tobacco, marijuana, LSD, cocaine, inhalants); and 4 single items assessing past 30-day alcohol use, binge drinking, being drunk or high in school, and tobacco use. Binge drinking and being drunk in school were not assessed for 6th graders, given the low rates of these behaviors at that age.

Community demographic information was gathered from 2000 census data aggregated at the school district level by the National Center for Educational Statistics. Variables used included percentage of households below the poverty line, percentage of households with nonwhite racial/ethnic backgrounds, population size, and percentage of female-headed households. The indicator of population density was based on census figures and obtained from publicly available data maintained by the Center for Rural Pennsylvania.

Data was collected from each CTC site regarding which programs they implemented, age groups or grades that participated, and dates of implementation. Each program was then checked to see if it was on the SAMHSA list of effective or model programs and if so, was coded as evidence-based. Grade cohorts were also coded at each school as potentially in the range or out of the range that could be affected by any of the CTC-sponsored programs.

Analysis: Feinberg et al., 2007: Separate analyses were conducted for the two waves of 2001 and 2003 PAYS data, and separately by grade (6th, 8th, 10th, and 12th) since risk and outcomes vary in a nonlinear fashion across grades. Analyses were conducted with statistical techniques that accounted for the clustering of students within school districts (multilevel regression models: SAS Proc Mixed). The indicator of community poverty was included in all analyses as a covariate; and a dichotomous variable representing condition (CTC vs. non-CTC) defined the program effect. Percentage of families living in poverty was used as a control (covariate) in all analyses.

Outcomes

Feinberg et al., 2007
Sample Adjustments and Equivalence: Analyses indicated that CTC districts did not significantly differ from non-CTC districts regarding percentage of nonwhite families, population size, population density, or percentage of female-headed family households. However, results indicate that school districts in CTC sites had a higher average proportion of poor family households (7.66%) compared to non-CTC districts (6.55%). The common region of support was examined and districts outside this region were eliminated. As a result, 11 CTC districts were eliminated that were higher on the poverty variable than any non-CTC districts (8 of these districts contributed data in 2001, and 7 districts contributed data in 2003). After eliminating these districts, the two groups no longer differed on level of poverty or other variables.

Results:
Overall, the pattern of results indicated that CTC school districts had lower levels of some risk factors and rates of substance use and delinquent behaviors than did non-CTC school districts. The results for all CTC sites compared to non-CTC sites indicated 4 significant comparisons favoring CTC in 2001 and 16 in 2003. Findings were particularly strong for the 6th and 12th grades in 2003. In addition, there were many fewer significant comparisons favoring non-CTC sites across the 2 years than would be expected by chance.

The first follow-up analysis included only those CTC grade cohorts that could have been directly impacted by an evidence-based model program. Because numerous CTC sites began implementing programs between 2001 and 2003, there were too few such cases in 2001 to analyze. For the 2003 data, analysis of cohorts most likely to be affected resulted in 22 significant comparisons favoring the CTC sites.

Examination of the distribution of significant effects indicated differential effects by grade and by risk factor/outcome. For the 2003 data, limited to CTC grade cohorts expected to be impacted by evidence-based programs, the greatest number of significant effects was found in the 6th grade. CTC influence on risk factors for the 6th grade included poor family supervision and discipline, friends' behaviors and attitudes, and the individuals own attitudes; analyses also indicated positive effects on each of the outcomes for 6th grade, with the exception of the composite drug use index. For the 6th through the 10th grades, analyses yielded consistent evidence of CTC influence on the outcome of delinquent behavior. And for the 12th grade, evidence of CTC effects was demonstrated for alcohol use and the multi-item measure of drug involvement.

Given that some of the variables were skewed, the data was reanalyzed utilizing models in SAS's PROC GLIMMIX procedure with alternate distributional assumptions. Poisson distributions were utilized and Ordered multinomial models were utilized where appropriate. Results were very nearly identical to the original models utilizing PROC MIXED with assumed normal distributions. In the 2003 expected impact analyses, 21 models yielded significant results with the re-analyses, compared to 22 in the original analyses.

Effect sizes were computed as odds ratios with multinomial models. These analyses were conducted for grades and risk factors/outcomes that had demonstrated significant differences between CTC and non-CTC sites in the 2003 expected impact subsample. Responses were recoded to facilitate analyses; for the variables analyzed, recoding yielded between 7 and 16 response levels. Because there were more than two levels of response for each variable, the resulting odds ratio was not interpreted as the likelihood of use vs. nonuse due to CTC as would be the case if typical logistic models had been used. Instead, the odds ratio resulting from this ordinal logistic model reflects the likelihood of a student in a non-CTC district reporting a higher level of risk or substance use than a student in a CTC district. Odds ratios for the models analyzed ranged from 1.12 to 1.36, with an average of 1.23.

Feinberg et al., 2010
As previous cross-sectional findings were open to the criticism that self-selection of communities into CTC may have biased results, this analysis utilized longitudinal data and tested for impact on change. The authors state that this design to a large extent removes the possibility that selection bias is responsible for the findings. It was not possible to track individuals over time due to the anonymous survey, thus the analyses concern changes in groups of individuals over time. For example, 6th graders responding to the 2001 survey were considered the same grade cohort as the 8th graders from the same school responding to the 2003 survey. Thus, subjects were nested within measurement periods which were nested within school district. In this study, each grade cohort at each school that was ever exposed to an evidence-based program was coded a "1" as having an "expected impact" by a universal EBP. All other grade cohorts (i.e., combining non-CTC and non-expected impact CTC grade cohorts) were coded as 0. Three-level hierarchical models were used to examine change in risk/protective factors, grades, delinquency and substance use over time.

There were no differences between CTC and non-CTC communities' grade cohorts in change of risk/protective factors, grades, and substance use. There was a significant difference between groups in delinquency, with youth in CTC communities, relative to control communities, demonstrating less growth in delinquency. In the models that tested Expected-Impact CTC grade-cohorts and comparison grade-cohorts (i.e., non-CTC grade cohorts and all other CTC grade cohorts), there were significant and beneficial intervention effects for risk/protective factors, academic grades, and delinquency, but not substance use.

Study 3

The two articles in this study used cross-sectional survey data for the same 24 randomized communities as in Study 1. Rhew et al. (2016) examined multiple survey years, while Van Horn et al. (2014) focused on the 2010 survey data.

Summary

Rhew et al. (2016) and Van Horn et al. (2014) also used data from the CYDS but examined a series of cross-sectional surveys between 2000 and 2010. Rhew et al. (2016) pooled the 2000 and 2002 assessments for baseline measures and the 2006 and 2008 assessments for follow-up measures and included 6th, 8th, and 10th graders. Sample sizes over survey years ranged from 4,647 to 5,077 for 6th graders, 4,491 to 4,984 for 8th graders, and 3,854 to 4,726 for 10th graders. Van Horn et al. (2014) examined the 2010 cross-sectional survey with samples of 4,674 eighth graders and 4,810 tenth graders.

Rhew et al. (2016), Van Horn et al. (2014):

Among 6th graders, there was a possible harmful effect as control participants had significantly better outcomes than treatment participants in:

• Antisocial behavior

In comparing change between 6th and 10th grade, the study found significant improvement in treatment communities than control communities for:

• Lifetime use of smokeless tobacco

For tenth-grade students in 2010, the study found lower levels in treatment communities than control communities for:

• Alcohol use

Evaluation Methodology

Design:

Recruitment: The data for this study was from the CYDS, a community-randomized controlled trial of 24 small towns in Colorado, Illinois, Kansas, Maine, Oregon, Utah, and Washington. The CYDS is discussed further in Study 1. The towns were selected from a larger sample of 41 communities participating in an observational study of the diffusion of science-based prevention strategies. Of these, 24 were eligible and consented to participate.

Assignment: The participating communities were matched within states according to population size, racial/ethnic composition, crime rate, and socioeconomic indicators. From each pair, one town was randomly assigned to the treatment condition the other to the control condition. Communities in the control condition did not receive any training in or funding for CTC activities from the CYDS project during the course of the study.

The repeated cross-sectional design compared all students in community schools at each survey year rather than followed individuals longitudinally. The surveys excluded students present at the start of the program who left the schools and included new students who entered the schools after the program start. All comparisons from baseline to follow-up thus included somewhat different samples of students.

Attrition: Assessments occurred at baseline (2000 and 2002) and follow-up (2006, 2008, and 2010). Across the years of the study, the response rate among 6th graders ranged from 79 to 87%, for 8th graders ranged from 72 to 84%, and for 10th graders ranged from 60 to 78%.

Sample: The sample was approximately half male (47.7 to 49.9%) and largely white (64.1% to 77.2%). The average age was 11.6 among 6th graders, 13.6 among 8th graders, and 15.6 among 10th graders.

Measures: Rhew et al. (2016) examined 9 measures of substance use and 12 measures total. Participants were asked to report use of alcohol, cigarettes, marijuana, and smokeless tobacco in the past 30 days and in their lifetimes. In addition, participants were asked whether or not they participated in binge drinking in the past 2 weeks. The study also included 2 measures of delinquency, based on whether or not the individual had sold illegal drugs, stole or tried to steal a motor vehicle, attacked someone, brought a handgun to school, or were arrested. In assessments after the baseline, 3 additional behaviors were added: stole something worth $5, purposely damaged or destroyed someone else's property, and shoplifted. The study recoded these responses into a count of the number of delinquent behaviors and a dichotomous report of any delinquent behavior. Finally, the study had one measure of change in risk and protective factors prioritized by the community.

Van Horn et al. (2014) created four latent-class measures of substance use and delinquent behaviors for the eighth-grade students: abstainer, experimenter, drug user, and problem behavior. For tenth-grade students, the latent-class analysis added an alcohol use measure to the other four measures.

Analysis: To examine the effects of CTC on youth outcomes, Rhew et al. (2016) used a two-stage ANCOVA approach. In the first stage, the study used individual-level data to estimate the adjusted prevalence of each problem behavior for a community in a given year, by grade. This was then used to create an adjusted pooled baseline level in 2000 and 2002 for each outcome and community and an adjusted pooled follow-up level in 2006 and 2008 for reach outcome and community. In the second stage, the study used a mixed effects model in which the adjusted post-intervention pooled community prevalence was regressed on study condition, the pooled baseline level for the community, percent free or reduced price lunches, and number of students in the community.

Rhew et al. (2016) also conducted analyses of changes in prevalence over time within a specific pseudo cohort followed from 6th grade to 10th grade. The study assumed that most students in 6th grade in 2004 were in 10th grade in 2008, creating the pseudo cohort. The analyses used for this pseudo cohort were similar to those above for the grade samples.

In examining cross-sectional data for 2010, Van Horn et al. (2014) used latent class analysis to assess the intervention impact on the latent classes of substance use/delinquency in the eighth and tenth grades.

Intent-to-Treat: Rhew et al. (2016) used all available data with multiple imputation for missing data. Van Horn et al. (2014) used full information maximum likelihood estimation to include participants with partial missing data. Only those "missing data on all variables" were excluded.

Outcomes

Implementation Fidelity: The study did not include any reports of implementation fidelity.

Baseline Equivalence: As discussed above, the communities were assigned in matched pairs based on population size, racial/ethnic composition, crime rate, and socioeconomic indicators. The study did not provide information on equivalence of baseline outcome scores of either communities or students.

Differential Attrition: Although there were no formal tests for attrition, the study examined differences between conditions on stable demographic characteristics at posttest. Program completers in the treatment group were more likely to be white as compared to the completers in the control group. The study did not discuss differential attrition in the outcome measures.

Posttest: Rhew et al. (2016) found one possible iatrogenic effect on antisocial behavior among 6th graders. Those in the treatment group self-reported significantly more antisocial behavior at posttest than the control group. None of the other outcomes were significant among 6th graders and none of the 12 outcomes were significant for either 8th or 10th graders.

In its analysis of a pseudo cohort, Rhew et al. (2016) found that of 12 outcomes, only lifetime use of smokeless tobacco saw significant improvement from 6th to 10th grade when comparing the treatment to control groups.

Using cross-sectional data from 2010 for eighth graders (n = 4,674) and tenth graders (n = 4,810), Van Horn et al. (2014) found one significant intervention effect in seven tests (see Table 2). Tenth graders in the intervention group reported significantly lower scores on a latent-class measure of alcohol use than those in the control group.

Long-Term: No long-term follow-up was reported.

Study 4

Summary

Chilenski et al. (2019) used a quasi-experimental, repeated cross-sectional design with propensity weighting to assess the effect of the intervention on substance use, delinquency, and depression in Pennsylvania students. Data were collected in six waves over a span of 11 years from students in grades 6, 8, 10, and 12, yielding responses from a total of 470,798 students in 388 school districts. Students reported on recent and lifetime substance use (alcohol, tobacco, marijuana, other illicit drugs) and delinquency (arrests, suspensions, selling drugs) as well as depressive symptoms.

Chilenski et al. (2019) reported that across six waves of data, compared to non-CTC school districts, students in CTC school districts reported significantly lower rates of:

• Substance use
• Delinquency
• Depressive symptoms

Evaluation Methodology

Design:

The program was evaluated in Pennsylvania using a quasi-experimental, repeated cross-sectional design with propensity score weighting at the school district level. Data were collected from participating school districts (N=418) in six waves over 11 years, with assessments occurring every other year for students in grades 6, 8, 10, and 12. After dropping school districts in Philadelphia and Pittsburgh because of limited participation in the youth survey and some other school districts because of missing identifiers, the final number of districts was 388. A total of 470,798 student-level responses were recorded.

Group assignment was reconsidered for each survey year. School districts were included in the intervention group if they had received any CTC services (program, policy, event, or activity) for any age group or grade level within the 12 months preceding data collection. Of the 388 school districts in the analysis sample, 51.6% ever received any CTC services.

For each data wave, inverse probability weighting was used to establish equivalence across the intervention and comparison groups. Weights were created using 27 district-level variables including aggregate indicators of family size, employment, education, demographic composition, population size, and marital status (see Table S4 in supplement).

Sample:

Over all data waves, approximately 53% of students reported lifetime alcohol use (24% recent), 27% reported lifetime cigarette use (12% recent), and 17.7% reported lifetime marijuana use (9% recent).

Measures:

Assessments occurred in 2001, 2003, 2005, 2007, 2009, and 2011 for students in grades 6, 8, 10, and 12 using self-reported items from the CTC Youth Survey. Substance use was measured separately for alcohol, cigarettes, and marijuana, with students reporting any lifetime use and any use in the past 30 days. Severe substance use was assessed with four items reporting recent substance use in school, recent binge drinking, and lifetime/recent use of any other drugs. Delinquency over the preceding 12 months was measured with five items reporting whether a student had been suspended from school, sold illegal drugs, stolen a vehicle, been arrested, and/or assaulted another person. Depression was assessed using the mean of four items (a=.88) measuring agreement with statements like, "sometimes I think that life is not worth it." To strengthen the validity of the measures, students who reported using a fictitious drug were dropped.

Analysis:

The effects of the intervention were evaluated with means comparison and multilevel regression models (students nested within year, year within district) that controlled for individual and district-level characteristics.

Intent-to-Treat: The authors stated that "Due to concerns regarding missing data and spillover effects, school districts that were served by a CTC in 1 year, but later not nominated were dropped for those particular years." School districts in Pittsburgh and Philadelphia (n=30) were dropped from analysis because they were not in areas served by CTC coalitions and, due to the urban setting, were not considered comparable to other school districts in the sample. Students were dropped from analysis if they responded affirmatively to a control item about using a fictitious drug.

Outcomes

Implementation Fidelity:

Not reported. Authors note that CTC was implemented in Pennsylvania in the late 1990s and that many programs and practices implemented in the early years would not meet current standards of evidence-based programming. As a result, any coalitions implementing non-evidence-based programs would fail to meet current standards for CTC implementation fidelity.

Baseline Equivalence:

No differences between conditions on sociodemographic measures after propensity weights were applied, but it was not possible to test for differences on outcome measures.

Differential Attrition:

No typical attrition due to design, but no tests were done for the high levels of missing data in student responses.

Posttest:

Across all data waves, compared to the comparison group, students in CTC intervention school districts had significantly improved outcomes in 12 of 16 tests. They were significantly less likely to report recent alcohol use, recent and lifetime use of cigarettes and marijuana, in-school substance use, binge drinking, and lifetime use of other illicit drugs. Intervention group participants were also significantly less likely to report being arrested (lifetime and past 12 months), school suspension within the past 12 months, and selling illegal drugs within the past 12 months. Adjusted odds ratios ranged from .85-.95. Additional results that included only districts using evidence-based programs in the CTC intervention group showed slightly stronger effects.

Long-Term:

Results indicate consistently lower levels of substance use in CTC school districts, but it appears that the program was still in operation during all six assessment waves and, despite the six waves of follow-up, the results do not constitute true long-term effects.

Study 5

Summary

Toumbourou et al. (2019) used a quasi-experimental, repeated cross-sectional design with no matching. Cross-sectional data were collected across 15 years. A total of 41,328 grade school students (mean grade = 8) reported on their alcohol use, tobacco use, cannabis use, and antisocial behaviors.

Toumbourou et al. (2019) reported that across 15 years of data collection, students in CTC communities, compared to non-CTC communities, reported less:

• Alcohol use
• Tobacco use
• Cannabis use
• Antisocial behavior

In addition, students in CTC communities, compared to non-CTC communities reported:

• A greater average composite of protective factors
• A reduced average composite of risk factors

Evaluation Methodology

Design:

Recruitment: This study evaluated the effect of CTC community intervention exposure compared to Australian youth population trends. Student data were collected as part of the National CTC Youth Survey, administered between 1999 and 2015. Participants in this survey were youth in selected grade levels (unspecified, though average grade was 8) from participating schools in 109 Australian localities (communities). Relevant education authorities and principles gave their consent for their school to be included in the sample. Students were recruited during school hours.

Assignment: CTC was evaluated in this study using a quasi-experimental, cross-sectional design across 15 years. The intervention group in this evaluation included the five of 109 localities that completed all five phases of the CTC process (n = 13,861). The localities differed in the number of times they had initiated a CTC cycle and when those cycles were carried out. Some localities had completed a CTC cycle during the collection period and then discontinued the program. Others had CTC programs ongoing at the end of the data collection period. The other 104 localities served as the comparison group (n = 27,467). There was no formal matching between the CTC communities and the comparison communities.

Assessments/Attrition: The National CTC Youth Survey was implemented no more than once per year during the data collection timeframe. The current study used data from 11 waves of CTC youth surveys (years unspecified). The survey was completed by students at their schools during class time. All data were collected from cross-sectional, aggregated school surveys, and therefore there was not attrition in the typical sense. Student participation in the surveys between localities ranged from 50-90% (average of 73%).

Sample:

Across all collections, students (N = 41,328) had an average age of 13.5 years (SD = 1.7), their average grade was approximately 8, and 51.7% were female. It is unclear as to whether students participated multiple times over the 15-year data collection period.

Measures:

The data for the four primary behavioral outcomes were drawn from 11 cross-sectional CTC Youth Surveys (which specific years had measurements was not disclosed). The CTC Youth Survey is a well-known and validated measure of behavioral outcomes and youth risk and protective factors. Students reported lifetime alcohol, tobacco, and cannabis use separately. The alcohol and cannabis measures were coded to have dichotomous outcomes (tobacco use was already assessed dichotomously). Antisocial behavior was assessed by asking participants to report the number of times in the last 12 months they had 1) carried a weapon, 2) sold illegal drugs, 3) stolen or tried to steal a motor vehicle such as a care or motorcycle, 4) attacked someone with the idea of seriously hurting them, and 5) been drunk or high at school. Response options ranged from never to ten or more times. Antisocial behavior was scored as the count across the five items averaged to range from 0 to 1.

In addition, students reported on eight risk factors and seven protective factors. The risk factors included community substance abuse, low community attachment, parent favorable attitudes toward substance use, poor family management, family conflict, low commitment to school, academic failure, and perceived substance use. The protective factors included community opportunities for prosocial involvement, family attachment, family opportunities for prosocial involvement, family rewards for prosocial involvement, school opportunities for prosocial involvement, school rewards for prosocial involvement, and emotional control. Each of these risk and protective factors consisted of multiple items that were aggregated. These aggregated factors were then combined to form an average composite risk factor score and an average composite protective factor score.

Analysis:

The effects of the intervention were evaluated with multilevel linear mixed effects models with students nested within municipal localities (communities). Behavioral outcomes and risk and protective factor composite scores were predicted by the interaction of the survey year annual trend estimate and CTC status (CTC community vs. non-CTC community). Year 1 estimates served as baseline outcome controls in linear trend models. All analyses controlled for clustering of students within communities, gender, school grade, urban location, government school, survey response rate, community disadvantage, child country of birth, Indigenous status, peer substance use, family antisocial attitudes, and community mobility.

Intent-to-Treat: It appears that all available data were used. However, missingness did vary between outcomes (range: 1% - 15%). In addition, missing data were included in the referent for three variables (government school, child country of birth, and Indigenous stats) and were imputed using regression from correlated variables for gender, peer substance use, family antisocial attitudes, and community mobility. The results using models with imputed data were similar to those using nonimputed data, so the results were presented including the imputed data.

Outcomes

Implementation Fidelity:

No quantitative measures of fidelity were provided. However, the authors stated that an independent process evaluation reported that the intervention communities successfully completed all five phases of the CTC process.

Baseline Equivalence:

At baseline, the CTC communities, relative to the non-CTC communities, had significantly higher levels of alcohol use, cannabis use, family antisocial attitudes, higher community mobility, and lower percentages of non-Australian births.

Differential Attrition:

No typical attrition due to the cross-sectional repeated measures study design.

Posttest:

Across the data collection time frame, aggregated data from students in CTC communities, relative to non-CTC communities, showed significantly stronger annual linear trend decreases in all four behavioral outcomes: alcohol use (adjusted OR = .94), tobacco use (adjusted OR = .97), cannabis use (adjusted OR = .96), and antisocial behaviors (unstandardized b = -.001).

In addition, students in CTC communities reported more protective factors (as assessed by the protective factors composite variable, unstandardized b = .007) and fewer risk factors (as assessed by the risk factors composite variable, unstandardized b = -.007) compared to students in non-CTC communities.

Long-Term:

The results were analyzed using annual linear trends of repeated measures cross-section data across the 15-year data collection period, which coincided with the treatment. Therefore, there were no long-term outcomes.

Study 6

Summary

Jonkman et al. (2015) used a quasi-experimental design with propensity score matching to examine 630 adolescents living in ten neighborhoods in five Dutch cities. Three cities randomly assigned neighborhoods and two cities non-randomly assigned neighborhoods to the intervention and control groups. The adolescents were assessed on substance use, anti-social behavior, and risk and protective factors at baseline and four years later at posttest.

Jonkman et al. (2015) found no significant effects of the intervention on any of the outcome measures.

Evaluation Methodology

Design:

Recruitment: The sample came from ten Dutch neighborhoods in five Dutch cities. In these neighborhoods, 3,368 adolescents aged 12-14 and their parents were contacted. The 785 adolescents who returned both the parental and participant consent forms comprised the baseline sample. The authors stated that the study sample was representative of adolescents in the communities in terms of age, gender, and ethnicity.

Assignment: Although designed as a randomized controlled trial, the study became quasi-experimental. The ten neighborhoods were matched in pairs within cities, based on population size, racial and ethnic diversity, economic indicators, and rates of crime and other problem behaviors of adolescents. Three of the cities agreed to use random assignment of the pairs to the intervention and control neighborhoods, but two cities non-randomly assigned the neighborhoods. Consent rates were low and differed across the five intervention and five control neighborhoods (24.3% and 16.3%, respectively). The final sample of 785 adolescents included 511 (70%) in the intervention condition and 274 (30%) in the control condition.

Given the non-random assignment in two cities and the low consent rate, the study used a quasi-experimental design with propensity score matching for analysis of the data. After matching, the total sample dropped from 274 control and 511 experimental subjects to 230 control and 400 experimental subjects (630 in total). Only adolescents who were comparable according to the propensity score analysis remained in the sample. The study did not discuss the predictors used in the matching.

Assessments/Attrition: Assessments occurred at baseline in 2008 and every year until the posttest in 2011, a four-year period in total. Over three quarters (75.5%) of the adolescents participated in all four waves of data collection, 11.6% in three waves, 7.7% in two waves, and 5.2% in only one wave. However, the multivariate analysis included only the first (baseline) and last (posttest) assessments.

Sample:

At baseline, the adolescents were ages 12-14, with about 55% female and 80% Dutch.

Measures:

All outcome measures were self-reported by the adolescents. The primary outcomes included three alcohol measures, one smoking measure, and one anti-social behavior measure. The secondary measures included 13 risk and protective factors covering family, school, peer group, and community domains. Only those risk and protective measures with Cronbach's alpha coefficients of .70 and higher were used.

Analysis:

After pre-processing the data using propensity score matching, the analysis used mixed models with dummy variables representing the city in which a subject lived. Intervention-by-time interaction terms assessed the program effect from baseline to posttest. The three levels of clustering included measurements, adolescents, and matched sets of communities (with measurement referring to baseline and posttest assessments). The models used "all the information of the adolescents who participated in the first wave of data collection" (p. 43). Both logistic and linear multilevel analyses were used, depending on the type of outcome. Bonferroni corrections were used to control for multiple significance tests.

For initiation of alcohol use and smoking, the analysis used survival models.

Missing Data Methods: The FIML estimation included participants with missing posttest data under the missing-at-random assumption.

Intent-to-Treat: The analysis used all participants with baseline data.

Outcomes

Implementation Fidelity:

The study didn't provide any formal measures of implementation fidelity, but the authors noted that the full program implementation "was hampered by the very small number of tested and effective preventive interventions available in the Netherlands." Also, implementation of the programs was delayed in some neighborhoods.

Baseline Equivalence:

Table 1c shows six significant differences in 25 tests before matching and two significant differences in 24 tests after matching. However, the tests include only sociodemographic measures, not outcomes.

Differential Attrition:

Attrition appeared to be similar across conditions. About 76.3% of the intervention adolescents and 74.1% of the control adolescents participated in all four waves of data collection. In addition, the authors stated that "Examination of the background variables among experimental and control adolescents over the years showed similar patterns of age, gender, ethnicity, and parental work status."

Posttest:

The results for the five substance use measures in Table 2 showed no significant intervention effects, and survival analysis of initiation to alcohol use and smoking also showed no significant interaction effects.

The results for the 13 risk and protective factors in Table 3 showed no significant effects, either for the whole sample or for any of the paired communities.

Long-Term:

It appears that the program was intended to last all four years of the study period, making the last assessment a posttest rather than a long-term follow-up.