Multisystemic Therapy® (MST®)

Study 1

On average, the treatment duration was 13.4 weeks, encompassing 33 hours of direct contact, and sessions ranged in duration from 15 to 90 minutes.

Summary

The study conducted a randomized controlled trial that assigned 84 youths at risk of out-of-home placement for serious criminal activity to an MST intervention group or a control group receiving usual youth services. Assessments at posttest and 2.4-year follow-up measured aggression, criminal activity, and arrests.

The study reported that, relative to the control group, the MST intervention group showed significantly

• Lower self-reported delinquency at posttest
• Less aggression with peers at posttest
• Greater family cohesion at posttest
• Fewer arrests and incarcerations at 59 weeks
• Longer time to rearrest through 2.4 years.

Evaluation Methodology

Design:

Recruitment: The sample consisted of 84 youths referred to the study in yoked pairs by the Department of Youth Services (DYS) in South Carolina. The primary referral criterion was DYS staff judgment that the adolescent was at imminent risk for out-of-home placement because of serious criminal activity (i.e., at least one felony arrest). Of the 96 youths referred to the study, 12 were excluded because the youth lacked a felony arrest (n = 2), MST was never implemented (n = 6; families moved or refused to participate), random assignment was violated (n = 2; MST was court ordered), or recidivism data were not available on the state's computerized system (n = 2).

Assignment: The Department of Mental Health randomly selected one youth of the yoked pair to receive MST (n = 43) and the other to receive the usual services delivered by the DYS (n = 41).

Assessment and Attrition: The program length varied by the client but lasted, on average, for 13.4 weeks. At posttest (Henggeler et al., 1992), self-report assessments were completed by 66% of the 84 sample youths. Usual-services families were evaluated at the same times as their yoked MST families (except when the youth was incarcerated, then the posttest assessment was conducted several weeks after release). In addition, archival data were obtained for all 84 sample youths at, on average, 59.6 weeks after the referral (Henggeler et al., 1992) and 2.4 years after referral (Henggeler et al., 1993).

Sample: The mean age of the sample was 15.2 years, and 77% were male, 56% were African American, 42% were Caucasian, and 2% were Hispanic American.

Measures: The main behavioral outcome measures came from archival records for arrest and incarceration. In addition, youths completed a self-report assessment of criminal behavior over the previous four months. Parents reported on youth psychiatric symptomology and social competence, but the ratings may be non-independent. Risk and protective factors included reports from youths and parents on family cohesion and adaptability, and perceptions of the adolescent's friendships on the dimensions of emotional bonding, aggression, and social maturity. The authors did not report on sample reliabilities for the scales but noted that measures obtained from both youths and parents were combined into a single scale to reduce error.

Analysis: Henngeler et al. (1992) used one-way analyses of variance and covariance (ANOVA), with baseline scores serving as covariates for the self-report measures. Henngeler et al. (1993) used survival analysis to model the probability of arrest.

Missing Data Method: The analysis used complete cases without imputation of FIML.

Intent to Treat: The study excluded six youths assigned to MST because they did not participate in the program and two youths who switched conditions. Both exclusions likely violated the intent-to-treat criterion.

Outcomes

Implementation Fidelity:

Not reported.

Baseline Equivalence:

Henggeler et al. (1992) stated that "The MST and usual services groups did not differ significantly on demographic variables or criminal history."

Differential Attrition:

There was minimal attrition for the archival measures. For the self-report data, Henggeler et al. (1992) stated, "Criminal histories and demographics of the participants completing both assessments are essentially the same as the larger sample." However, 77% of the families in the MST condition and 56% of families in the control condition completed the self-report assessments. Based on the overall attrition rate plus the difference in attrition rates across conditions, the study meets neither the cautious nor optimistic standard of What Works Clearinghouse and likely indicates attrition bias in the results. 

Tests of attrition in Henggeler et al. (1992) showed families who completed both assessments, compared to those who did not, were more likely to be African American, to have participated in the MST condition, and to have reported higher maternal symptomatology, more structured family relations, and greater social competence in the youth. Henggeler et al. (1992) noted, however, that "the variables on which attriters and nonattriters differed did not contribute to the outcome."

Posttest:

Table 1 (Henggeler et al., 1992) shows that the intervention group, relative to the control group, has significantly fewer arrests, fewer incarcerations, and low self-reported delinquency. For risk and protective factors, the intervention group had significantly higher ratings of family cohesion and significantly lower ratings of aggression with peers. Effect sizes were .45 for rearrests, .62 for incarceration, .55 for self-reported delinquency, .56 for family cohesion, and .34 for aggression with peers.

Tests for moderation showed that MST was equally effective with youths of different gender and ethnic backgrounds and with families possessing different levels of cohesion.

Long-Term:

A 2.4-year follow-up (Henggeler et al., 1993) showed that MST significantly extended the time to arrest for the intervention youths (56.2 weeks and 61% arrested) relative to the usual services group (31.7 weeks and 80% arrested).

Study 2

The study presented the results from two evaluations. The first, the Missouri Delinquency Project, is described in Study 3. The second, the Family and Neighborhood Services Project, is described here; the MST program lasted, on average, for four months, with 36 hours of direct contact.

Summary

Henggeler et al. (1991) conducted a randomized controlled trial that assigned 47 juvenile offenders to an MST intervention group or a control group receiving usual youth services. Assessments at posttest measured substance abuse and delinquency.

Henggeler et al. (1991) found that, relative to the control group, the MST intervention group reported significantly

• Lower soft drug use at posttest.

Evaluation Methodology

Design:

Recruitment: The sample came from 47 juvenile offenders in South Carolina who had committed a violent or non-violent felony and were at risk of imminent out-of-home placement.

Assignment: The study randomly assigned the offenders to the intervention group (n = 28) or to a control group (n = 19) that received usual services from the Department of Youth Services.

Assessments/Attrition: The posttest assessment occurred at the end of therapy. The authors provided no information on attrition, even though the uneven sample sizes for the conditions suggest the loss of participants after randomization.

Sample:

The mean age of the sample was 15.1 years, 72% were male, 74% were African American, and 26% were Caucasian.

Measures: 

Self-reported substance abuse was measured with the soft drug use and hard drug use subscales of the self-report delinquency scale in the National Youth Survey.

Analysis:

The analysis used ANOVA with baseline outcomes as covariates.

Missing Data Methods: The analysis appeared to use complete cases.

Intent to Treat: The study reported only the final sample size, without information on the loss of participants. The different sizes of the conditions suggest non-participants may have been dropped and the intent-to-treat criterion may have been violated.

Outcomes

Implementation Fidelity:

In the MST condition, 89% of the referred families participated in treatment.

Baseline Equivalence:

Using the analysis sample of 47, the authors reported finding no significant between-condition differences in "demographic characteristics."

Differential Attrition:

The study reported only the final sample size, without information on the loss of participants. Baseline equivalence in demographic characteristics suggests that differential attrition did not greatly bias the sample. However, the different sizes of the conditions suggest potential bias from differential attrition.

Post-test:

Self-reported soft drug use was significantly lower at posttreatment for the youths in the MST condition than for the youths in the control condition. Differences between samples for hard drug use could not be tested due to a very low base rate (i.e., only three youths reported such use).

Long Term:

Not examined.

Study 3

Summary

Borduin et al. (1995), Schaeffer and Borduin (2005), Sawyer and Borduin (2011), and Wagner et al. (2014) conducted a randomized controlled trial that assigned 176 juvenile offenders ages 12-17 to an MST intervention group or an individual therapy control group. Assessments at posttest through a 21.9-year follow-up measured problem behavior, arrests, and family relations.

The reports found that, relative to the control group, the MST intervention group showed significantly

• Decreased parent-reported youth behavior problems
• Improved family relations
• Fewer arrests
• Fewer arrests of siblings.

Evaluation Methodology

Design:

Recruitment: The sample included juvenile offenders (ages 12-17) and their families who were referred to the project by juvenile court personnel from 1983-1986. The juvenile offenders had at least two arrests, currently lived with at least one parent figure, and showed no evidence of psychosis. The study dropped 24 families who completed the baseline assessment but refused services, leaving 176 participants.

Assignment: The 176 families who completed the baseline assessment and agreed to participate in treatment were randomly assigned to receive either MST (n = 92) or individual therapy (n = 84).

Assessments/Attrition: Assessments occurred at baseline and posttest (after completion of therapy). A total of 126 families completed both assessments (72% of the 176 randomized participants). In addition, follow-up assessments using police and court records of adolescent criminal activity were conducted, on average, four years, 13.7 years, and 21.9 years after treatment had been completed. The follow-up data had no attrition. However, for participants who had moved out of state (6.2% at the 13.7-year follow-up and 15.9% at the 21.9-year follow-up), data were gathered through the latest date for which the youth could be confirmed to live in the state.

Sample:

The juvenile offenders had an average age of 14.8 years; 67% were male; 70% were White, and 30% were African American; 65% came from families characterized by low socioeconomic class; and 53% lived with two parental figures. By the 21.9-year follow-up, the participants averaged 37.3 years old.

Measures:

Borduin et al. (1995) relied on youth self-reports for behavioral measures of psychiatric symptomology, mother reports for measures of youth behavior problems, and mother and teacher reports for measures of peer relations, including peer aggression. Risk and protective measures of family functioning and family interactions came from mothers, youths, and observations of video-recorded family discussions.

Juvenile court, local police, and state police records were used to obtain data on post-probation arrests. Researchers unaware of condition assignment compiled figures from both juvenile and adult records. The measures were limited to arrests within the state of Missouri. Related measures included the total number of days that a participant convicted of an adult crime was sentenced to serve in an adult correctional facility and the total number of days that a participant convicted of an adult crime was sentenced to a term of probation. Sawyer and Borduin (2011) added measures of civil suits involving family instability and financial problems, and they limited the measures of arrests to those resulting in a conviction.

Analysis:

Borduin et al. (1995) used repeated measures multivariate analyses of variance and ANOVAs to examine changes from baseline to posttest and survival analysis to examine arrests through the four-year follow-up. For the 13.7-year follow-up, Schaeffer and Borduin (2005) used survival analysis and Tobit analysis for censored outcomes such as the number of arrests and days incarcerated. Given the truncated follow-up period for those moving out of state, the models controlled for differences in the length of participants' follow-up periods. Sawyer and Borduin (2011) used odds ratios, survival analysis, and zero-inflated Poison regression.

Missing Data Method: The posttest analysis used complete cases, without imputation or FIML for missing data, while the follow-up analyses had no missing data or used complete cases.

Intent to Treat: The posttest analysis (Borduin et al., 1995) used all participants with complete data. The study dropped referrals who refused the services but appeared to have done so before randomization. The study also examined results for program completers and dropouts but did so as a supplement to the intent-to-treat analysis. In contrast, Borduin et al. (1995) and Schaeffer and Borduin (2005) and Sawyer and Borduin (2011) used the full randomized sample for some analysis, and the subsample with complete data for other analyses.

Outcomes

Implementation Fidelity:

Of the 92 participants assigned to MST, 84% attended at least seven sessions. Checklists completed by therapists revealed that, among the 84%, none of the participants received interventions in only one system, 33.8% received interventions in two different systems (most often family and school), and 66.2%received interventions in three (most often family, school, and peer) or more systems.

Baseline Equivalence:

Tests using the randomized sample of 176 revealed no significant differences between participants in the two treatment conditions on criminal history and demographic characteristics as well as on measures of individual adjustment, family relations, and peer relations (Borduin et al., 1995, p. 572). Tests using the posttest analysis sample of 126 found that the conditions did not differ on any demographic variable or measure of criminal history (Borduin et al., 1995, p. 570).

Differential Attrition:

First, note that the criminal behavior measures did not have attrition.

For the posttest assessment (Borduin et al., 1995), some evidence of differential attrition emerged. Attrition rates were 24% in the intervention group (based on 70 completers and 92 randomized participants) and 33% in the control group (based on 56 completers and 84 randomized participants). The overall attrition rate combined with the difference in condition attrition rates do not meet the cautious or optimistic WWC standards. However, the study reported on a variety of main-effect and interaction tests, including tests for baseline equivalence in the analysis sample, that showed no evidence of differential attrition.

The analyses of Schaeffer and Borduin (2005) had no attrition. Some of the analyses in Sawyer and Borduin (2011) had attrition, but they reported that "There were no differences in the pretreatment criminal histories or demographic characteristics of participants included in the follow-up versus participants considered lost to follow-up."

Posttest:

Based on survey self-reports and observations of family interaction, Borduin et al. (1995) found a significant intervention effect on one behavioral outcome: The intervention youths showed greater improvement in mother-reported problem behavior than the control youths. The intervention group also showed greater improvement than the control group on several risk and protective factors: symptomatology in parents, family cohesion, family adaptability, family supportiveness, and family conflict-hostility.

Long-Term:

Four-Year Follow-Up: Borduin et al. (1995) found that the intervention youths were arrested significantly less often (26% versus 71%) and for less serious crimes than the control group. The survival analysis further showed a significantly longer time to arrest for the intervention than the control group.

13.7-Year Follow-Up: Schaeffer and Borduin (2005) found that the intervention group relative to the control group had significantly fewer rearrests for any offense (OR = 4.25), any violent offense (OR = 2.57), any nonviolent offense (2.63), and any drug offense (3.33). Reflecting the different rearrest rates, the survival functions for the two groups were significantly different, with a longer time to arrest for the intervention group. In addition, using a control for the variation in the length of the follow-up period and a subsample of 165 participants who lived in the state during the follow-up period, analyses found that the intervention group had significantly fewer days sentenced to adult confinement than the control group.

21.9-Year Follow-Up: Sawyer and Borduin (2011) found that the treatment group had significantly better outcomes for four of the six arrest measures: felonies (OR = 2.27), violent felonies (OR = 4.08), nonviolent felonies (OR =1.97 ), and civil suits involving family instability (OR = 2.08 ). The survival analysis showed a longer time to arrest for the intervention group than the control group. In addition, using a control for the variation in the length of the follow-up period and a subsample of 148 participants who lived in the state during the follow-up period, analyses found that the intervention significantly fewer arrests for misdemeanors.

Moderation: Tests in Borduin et al. (1995), Shaeffer and Borduin (2005), and Sawyer and Borduin (2011) for potential moderators (age, race, social class, gender, pretreatment arrests) showed no significant differences in program effects.

25 Year Sibling Follow-Up (Wagner et al., 2014):

To examine the general effects of MST beyond the youth participants, the 25-year follow-up measured arrest rates and incarceration times for the closest-in-age sibling of those targeted in the original MST study.

Evaluation Methodology

Design:

Recruitment: Of the 176 randomized juveniles included in the original study, 129 had siblings in the home during the period of intervention. These 129 closest-in-age siblings, rather than the original sample, served as subjects in the 25-year follow-up.

Assignment: The original subjects, but not the sibling subjects, were randomized to the treatment and control groups. Among the 129 siblings participating, 67 (51.9%) belonged to the treatment group and 62 (48.1%) to the control group.

Attrition: The study obtained publicly available Missouri court records of the siblings. Of the 129 participants, 19 (14.73%) appeared to have left the state, lacked data on court records, and were designated as lost to follow-up.

Sample:

Of the selected participants, 60% were the younger sibling of the MST participant, while 40% were older siblings. Half of the siblings included in the study were male. The majority of participants were white (86.4%).

Measures:

The study used publicly available adult (age 17+) criminal records. Researchers coded crime classification (felony or misdemeanor) and date of arrest. In addition, the study used the number of days sentenced to incarceration or probation. The inability to obtain data for states other than Missouri might limit the measures, as might the inability to obtain data on juvenile offenses.

Analysis:

Based on the publicly available criminal records, the study analyzed the relative odds of arrest of control and treatment group participants. In addition, the study reported the cumulative survival function for arrests. Finally, the number of arrests and years sentenced were analyzed using a Zero Inflated Poisson Regression, which evaluates the impact of treatment condition on the number of posttreatment arrests and years sentenced to incarceration or probation.

Missing Data Method: The analysis used complete cases without imputation or FIML.

Intent-to-Treat: The analysis excluded the 19 individuals who were lost to follow-up because of residence outside the state but otherwise used all available data.

Outcomes:

Implementation Fidelity:

None of the sample siblings received treatment.

Baseline Equivalence:

A comparison of the treatment and control siblings for the analytic sample of 110 subjects showed no significant differences on demographic variables, but the study lacked baseline measures for other variables.

Differential Attrition:

No baseline differences in condition or demographic variables were reported between those located and those lost to follow-up, but the study lacked baseline measures for other variables. The attrition rate of 10% for the intervention group, 19% for the control group, and 15% overall attrition rate of 15% meet the WWC optimistic standard but not the cautious standard.

Posttest and Long-Term:

• Siblings in the control group were significantly more likely to have been arrested at least once as compared to siblings in the treatment group (OR = 3.36).
• A log-rank test of the survival functions for the treatment and control group showed that the treatment group was at a significantly lower risk of arrest (p=.004) and misdemeanor arrest (p=.02) during the follow-up period than the control group.
• The Zero Inflated Poisson Regression analysis included the odds estimate for any arrests and rate estimates for number of arrests. The treatment significantly reduced the odds of arrest for any crime, any felony, and any misdemeanor but did not influence the number of arrests.
• The Zero Inflated Poisson Regression also examined adult sentencing, finding that the treatment significantly reduced the odds of incarceration and probation and the years of the sentence for incarceration and probation.

Study 4

Summary

Henggeler et al. (1997) conducted a randomized controlled trial that assigned 155 juvenile offenders to an MST intervention group or a usual-services control group. Assessments at posttest and 1.7-year follow-up measured emotional functioning, re-offending, and family relations.

Henggeler et al. (1997) found that, relative to the control group, the MST intervention group showed significantly:

• Better emotional-behavioral functioning
• Fewer days incarcerated.

Evaluation Methodology

Design:

Recruitment: Participants were 155 violent or chronic juvenile offenders and their primary caregivers. The adolescents (a) were between  11 and 17 years of age, (b) had committed a serious criminal offense or had at least three prior criminal offenses other than status offenses, and (c) were at imminent risk of being placed outside the home because of serious criminal involvement. The youths were referred to the study at two judicial sites in South Carolina, each with three counties encompassing both rural and urban areas.

Assignment: Families were randomly assigned to the intervention group (n = 82) or a usual services control group (n = 73). The authors temporally yoked intervention and control families to sync the time of assessment (see page 824). It was not possible to yoke nine intervention families, but yoking did not appear to affect the randomization or analysis.

Assessment/Attrition: Assessments occurred at baseline, posttest, and a 1.7-year follow-up from the project end date. By posttest, 9.7% of the families had dropped out of the study. The follow-up data appeared to have no attrition.

Sample:

The youth participants averaged 15.22 years of age; 81.9% were male; 19.4% were White, and 80.6% were African American. Approximately 50% were from two-parent households, and the median family income was between $5,000 and $10,000 per year.

Measures:

The outcome measures at posttest came from youth and independent parent reports of emotional-behavioral functioning, delinquency, and family relations. The scales had good reliability. The outcome measures at follow-up came from state records on juvenile offending and incarceration.

Analysis:

For posttest outcomes, Henggeler et al. (1997) used repeated-measures ANOVAs with condition-by-time interaction terms that controlled for baseline outcomes. For long-term offending outcomes, the ANOVAs did not control for baseline outcomes.

Missing Data Method: The posttest analysis used complete cases, while the 1.7-year follow-up used all cases.

Intent to Treat: The analysis used all participants or all participants with complete data.

Outcomes

Implementation Fidelity:

The treatment for the MST groups averaged 122.6 days for site 1 and 116.6 for site 2. The study measured six components of intervention adherence but did not report mean levels.

Baseline Equivalence:

No tests reported.

Differential Attrition:

Henggeler et al. (1997) reported that mothers from the dropout group were significantly more educated than mothers from the completer group. No other between-group differences in demographic or dependent variables were observed. The differences in condition attrition rates combined with the overall attrition rate show little evidence of bias under either the WWC cautious or optimistic standard.

Posttest:

For behavioral outcomes, the top panel of Table 2 (Henggeler et al., 1997) shows two significant effects in seven tests. Youths in the MST condition reported significantly better emotional and behavioral functioning and significantly fewer days incarcerated than the control group. For risk and protective factors related to family relations and peer relations, the bottom panel of Table 2 shows no significant effects in 15 tests.

Mediation: Huey et al. (2000) examined 57 of the MST participants who had data on intervention adherence (referred to as the Diffusion Project). The analysis did not use the control group. The authors summarized the results by saying: "Overall, the Diffusion models support the hypothesis that therapist adherence to MST principles contributes both directly and indirectly to reductions in delinquent behavior."

Long-term:

At the 1.7-year follow-up (Henggeler et al., 1997), the intervention group had significantly fewer days incarcerated than the control group but did not differ significantly in arrests.

Study 5

This study evaluated an early version of Multisystemic Therapy called Family-Ecological Treatment.

Summary

Henggeler et al. (1986) used a quasi-experimental design with matching that examined 163 youths who were non-randomly assigned to three groups: an MST intervention group for juvenile offenders, an alternate family intervention for juvenile offenders, or a comparison group of non-offenders. Assessments at posttest measured conduct problems, anxiety, and aggression.

The study found that, relative to the alternative services comparison group, the MST intervention group showed significant

• Decreases in behavior problems at posttest
• Improvements in family relations at posttest.

Evaluation Methodology

Design:

Recruitment: The initial sample of youths and families came from three groups. First, 116 families of juvenile offenders were referred to the family-ecological (or MST) treatment from the Memphis-Metro Youth Diversion Project. The 87 who completed treatment were used in the study. Second, 40 juvenile offenders and their families were selected from those referred by the Memphis-Metro Youth Diversion Project to other mental health agencies for services. These subjects were chosen for inclusion in the study based solely on a demographic makeup that was similar to that of the family-ecological treatment group. The 26 who completed treatment were used in the study. Third, 50 non-pathological, relatively well-adjusted adolescents and their families were recruited at local high schools. Those selected matched the demographic characteristics of the treatment families.

Assignment: The quasi-experimental design did not use random assignment but instead selected participants who matched on demographic variables and pretreatment arrest histories.

The authors noted some potential confounds. The therapists of the alternative treatment adolescents had considerably more experience than the family-ecological therapists, a difference that favored the alternative treatment group. Also, the alternative treatment families received an incentive for completing the posttest, while the other groups did not.

Assessments/Attrition: Assessments occurred immediately before treatment and within three weeks after the treatment ended. The posttest for the normal families was conducted between two and six months following baseline (average of three months). This waiting period was intended to reflect a time frame similar to that of the families in treatment. Of the 163 study participants, 124 (76%) completed the posttest.

Sample:

The sample averaged 14.8 years of age; 65% were black; 75% were lower class on the Hollingshead socioeconomic scale; 62% were father-absent; and 84% were male.

Measures:

Child psychopathology and family relations outcomes came from youth and parent surveys. The measures had been validated by others, but the authors did not report reliability or validity information for the study sample. Other measures of family relations came from video observations of youth and parent discussions that were coded by raters blind to condition. Interrater reliability was high.

Analysis:

The analysis used multivariate and univariate ANOVAs to examine changes from baseline in the outcomes.

Missing Data Method: The analysis used complete cases without imputation or FIML.

Intent to Treat: The analysis violated the intent-to-treat criterion by dropping participants in the two treatment groups who did not complete therapy

Outcomes

Implementation Fidelity:

The authors noted only that the style and quantity of the family-ecological treatment was based on the family's needs and varied widely (from 2 to 47 hours, mean = 20 hours).

Baseline Equivalence:

The authors stated that "demographic and psychometric characteristics of the three groups showed no significant differences based on univariate analyses of variance (ANOVAs) and nonparametric analyses." They also stated that "the arrest histories of the adolescents in the family-ecological treatment condition did not differ from those of the adolescents in the alternative treatment condition."

Differential Attrition:

The study did not test for differential attrition but presented figures on attrition by group. The attrition rates of 34% for the family-ecological treatment group, 12% for the alternative treatment group, and 12% for the control group indicate large differences and potential for bias. The overall attrition rate combined with differences in attrition rates across conditions meets neither the WWC cautious nor optimistic standard.

Posttest:

Table 1 shows significantly greater decreases for the family-ecological condition than the alternative treatment condition in measures of youth conduct problems, anxiety-withdrawal, immaturity, and socialized aggression. No significant multivariate interaction effects emerged in self-reported family relations. However, observational ratings indicated significant positive change among the family-ecological group (e.g., the mother-adolescent and marital dyads were warmer and more affectionate, and the adolescent was more actively involved in family discussions after treatment).

Long-Term:

Not examined.

Study 6

Henggeler et al. (1999, 2002) provided the key evaluations of program effects on crime and substance use. Brown et al. (1999) examined school attendance, Schoenwald et al. (1996) examined program costs, and Huey et al. (2000, Study 2) examined mediation.

Summary

The study conducted a randomized controlled trial that assigned 118 youths with diagnosed substance abuse to an MST intervention group or a usual-services control group receiving outpatient substance use treatment. Assessments from posttest to a four-year follow-up measured drug use, criminal activity, and out-of-home placement.

Henggeler et al. (1999, 2002) and Brown et al. (1999) found that, relative to the control group, the MST intervention group showed significantly

• Lower substance use at posttest and four-year follow-up
• Better school attendance at six months
• Fewer aggressive crimes at four-year follow-up.

Evaluation Methodology

Design:

Recruitment: The sample included 118 adolescents ages 12-17 years who were recruited from the Department of Juvenile Justice in Charleston County, South Carolina. Eligible youths met diagnostic criteria for substance abuse or dependence, had formal or informal probationary status, and lived with at least one parent figure. Out of 140 screened adolescents who met the eligibility criteria, 118 (84%) agreed to participate in the study.

Assignment: The study randomly assigned the participating families to the intervention condition (n = 58) or a usual-services control condition (n = 60). Control youths were referred by their probation officer to receive outpatient substance abuse services from the local office of the state Substance Abuse Commission. Participation was low, however. About 78% of the control families received no treatment.

Assessments/Attrition: Assessments occurred at baseline, posttest, six-month follow-up, and four-year follow-up. In Henggeler et al. (1999), measures of drug use and crime obtained from archival records had no attrition, while self-report measures were completed by 97% of the families at posttest and 92% at the six-month follow-up. Brown et al. (1999) did not report directly on missing data for school attendance measures, but page 88 reports a sample size of 84, or 71% of the randomized sample of 118. Henggeler et al. (2002) reported a 68% completion rate at the four-year follow-up.

Sample:

The average age of the sample was 15.7 years at the time of referral, with 79% male, 50% African American, 47% Caucasian, 1% Asian, 1% Hispanic American, and 1% Native American. Based on socioeconomic measures, the sample was relatively disadvantaged.

Measures:

Henggeler et al. (1999) and Henggeler et al. (2002) used self-reports, parent reports, urine tests, and archival records to examine drug use, criminal activity, out-of-home placement, and psychiatric symptoms. Drug use was assessed through self-reports and urine toxicology screenings. Criminal activity was measured through self-reports and computerized arrest records. Days in out-of-home placement were documented through the monthly service utilization survey.

Brown et al. (1999) examined three measures of school attendance data: school records, youth and parent-reported attendance from the Child Behavior Checklist, and the monthly service utilization survey completed by caregivers.

Analysis:

Henggeler et al. (1999) and Brown et al. (1999) used repeated-measures ANOVA models that included the baseline assessment or analysis of covariance models that controlled for baseline drug use. Henggeler et al. (2002) first used multivariate analyses of covariance (MANCOVAs) with youth age and baseline marijuana use (but not the baseline outcomes) as covariates. If significant, the MANCOVA was followed by one-way ANCOVAs on each measure. In addition, chi-square analyses were conducted on the biological indicators of marijuana and cocaine use as a measure of abstinence (i.e., no positive test versus any positive test). Schoenwald et al. (1996) focused on measuring costs and reported group means without significance tests.

Missing Data Method: The study used complete-case analyses, without imputation or FIML

Intent to Treat: The study used all participants with complete data.

Outcomes

Implementation Fidelity:

Nearly all of the MST families (98%) completed a full course of treatment. However, Henggeler et al. (1999) noted that the lack of therapist supervision by the program developers lowered implementation quality. To quote, "The modest results in this study may be due to the difficulty in transporting MST from the program developers to the supervisors and therapists implementing the program."

Baseline Equivalence:

Using the full randomized sample, Henggeler et al. (1999) found that MST youths were equivalent to control youths on sociodemographic and outcomes measures with one exception. MST youths reported higher rates of drug use before treatment than control youths. Using the sample of youths with baseline data, Brown et al. (1999) added that the conditions did not differ significantly at baseline on the measure of school enrollment.

Henggeler et al. (2002) tested for baseline equivalence using the analysis sample of 80 completers. The tests examined demographic characteristics (i.e., gender, race, age), illicit drug use, and criminal behavior. The results indicated two significant differences. MST participants were older and reported more frequent marijuana use. Because of the differences, baseline age and marijuana use were included as covariates in the outcome analyses.

Differential Attrition:

Henggeler et al. (1999) did not report formal tests of differential attrition but had no attrition for some measures and low attrition overall rates of 3% at posttest and 7% at the six-month follow-up for other measures. The differences in condition attritions rates was also small and met both the WWC cautious and optimistic standards. Brown et al. (1999) did not report on attrition.

Henggeler et al. (2002) tested for baseline differences between completers and dropouts and reported no significant results for treatment condition, gender, race, presence of DSM-III-R internalizing and externalizing disorders, substance abuse or dependence status, illicit drug use, and aggressive and property crimes. However, the high overall attrition rate combined with the difference in condition attrition rates does not meet either the WWC cautious or optimistic standard. Also, tests for baseline equivalence using the analysis sample produced two significant differences.

Posttest:

Henggeler et al. (1999, Table 2) found no significant treatment effects for drug use, delinquency, or arrests. The text reported that the intervention group had fewer incarcerated youths and shorter time incarcerated but did not mention significant tests. Moderation tests found intervention benefits on alcohol and marijuana use for females and younger participants at posttest.

Brown et al. (1999) found a significant condition-by-time interaction that showed a greater increase in school attendance among intervention youths than among control youths.

Schoenwald et al. (1996) performed a monetary cost analysis through the six-month follow-up. The total cost of services in the US condition during the 11-month period was $198,729. Based on the clinical outcomes in this study - a 46% reduction in incarceration days and a 64% reduction in hospitalization/residential treatment - the higher costs of implementing the MST program are nearly offset by the savings in incarceration costs.

Huey et al. (2000) examined 54 MST participants with complete pretest and posttest data (referred to as the Charleston Drug Abuse Project). The analysis did not use the control group. The mediation model showed that "improvement in family cohesion and monitoring was associated with decreased affiliation with deviant peers, which, in turn, was associated with a reduction in delinquent behavior." Also, therapist adherence contributed indirectly to reductions in delinquent behavior."

Long-Term:

In Henggeler et al. (2002), Table 1 shows three significant effects in 10 tests. MST participants had significantly fewer self-reported aggressive crimes, convictions for aggressive crimes, and positive urine tests for marijuana. Moderator analyses showed that the impact of treatment did not vary as a function of demographic characteristics, comorbid psychopathology, or initial levels of drug use and criminal behavior.

Study 7

The Norwegian program implementation followed the MST treatment manual with no major modifications.

Summary

The study conducted a randomized controlled trial that assigned 100 Norwegian youths with serious antisocial behavior to an MST intervention group or a control group receiving child welfare services. Assessments at posttest and two-year follow-up measured problem behavior, delinquency, and out-of-home placement.

The study found that the MST intervention group, relative to the usual-services control group, showed significantly:

• Decreased internalizing symptoms at posttest
• Increased social competence at posttest
• Decreased out-of-home placements at posttest
• Lower delinquency at the two-year follow-up
• Fewer behavior problems, internalizing, and externalizing at the two-year follow-up.

Evaluation Methodology

Design:

Recruitment: The sample included 100 adolescents and their families from four counties in Norway who were referred to treatment for antisocial behavior by Municipal Child Welfare Services. For reasons not related to the MST program, the therapists at one site resigned, and the site was replaced by one selected from the next group of teams to be trained. The adolescents were between 12 and 17 years old, exhibited serious problem behavior, and had parents who were sufficiently involved and motivated to start MST. All families approached for the study agreed to participate.

Assignment: A weighted randomization procedure was utilized to assign families to the treatment or control condition, with each family having a 6/10 chance of receiving MST and a 4/10 chance of receiving the usual services (e.g., institutional placement, home treatment). This resulted in 62 families being assigned to MST and 38 families to the control condition. The authors noted that the community services available to the control group were much more comprehensive and treatment-oriented than those typically available to juvenile offenders in previous MST research, creating a more stringent test for program effects.

Four families dropped out of the MST program early in the treatment but were replaced with new families. Also, Ogden and Hagen (2006) dropped one site with 25 participants because it did not collect implementation data and was deemed to have not implemented the program. For the truncated sample of 75, the MST group had 46 families and the control group had 29 families.

Assessments/Attrition: Assessments were conducted at study entry, after termination of MST treatment (approximately 6 months after intake), and 18 months after treatment (or approximately two years after intake). At the posttest, four families withdrew, resulting in a 96% retention rate (or a 92% attrition rate when counting the MST dropouts who were replaced). At the follow-up, the study dropped one site, leaving 75 potential participants, of which 69 (92%) completed the assessment. In addition, teacher data was available for only 29 youths (39%), in part because many no longer attended school.

Sample:

The baseline sample consisted of 63 boys and 37 girls, who averaged 14.95 years of age. Nearly all (95%) of the caregivers had a Norwegian background. Thirty-nine percent of the sample had been previously placed out of the home, 54% had a history of running away from home, 30% had been suspended from school, and 90% had a history of school truancy.

Measures:

Outcome measures came from parents, teachers, and youths. All three rated youth internalizing, externalizing, and social competence. Youths provided a measure of self-rated delinquency. The Family Adaptability and Cohesion Evaluation Scales-III (FACES-III) were completed by both caregivers and adolescents. This study also assessed out-of-home placement. The study scales showed high reliability.

Analysis:

Ogden and Halliday-Boykins (2004) used ANOVA and chi-square tests with controls for site and baseline outcomes, except for the outcome measure of out-of-home placement. Ogden and Hagan (2006) used ANCOVA with covariates for age, gender, and baseline outcomes, unless unrelated to the outcome.

Missing Data Method: The complete-case analyses did not use imputation or FIML. The one exception involved missing data for teacher reports in Ogden and Hagan (2006). The expectation-maximization procedure was used for missing teacher reports.

Intent to Treat: Ogden and Halliday-Boykins (2004) replaced four families that dropped out of MST, which may violate the intent-to-treat criterion. Ogden and Hagen (2006) violated the intent to treat criterion by dropping one site with 25 participants because of uncertain implementation quality.

Outcomes

Implementation Fidelity:

Ogden and Halliday-Boykins (2004) noted only that MST participants reported marginally greater satisfaction with the treatment than control participants.

Baseline Equivalence:

A comparison of the two groups at baseline showed control caregivers were more likely than MST caregivers to be divorced, and MST caregivers were more likely to be married to someone other than the child's biological parent. The authors noted that "this difference had no moderating effect on the outcomes"

Differential Attrition:

Ogden and Halliday-Boykins (2004) reported that the attrition analysis was not possible because the government-approved informed consent allowed dropouts to have their data expunged from research records.

Ogden and Hagen (2006) reported "no significant differences between the families that withdrew from the research project and the remaining participants on demographic or treatment assessment scores on psychosocial variables." Based on the truncated sample of 75, attrition overall was 8%, and the difference between conditions was 4%, which meets the WWC's cautious and optimistic standards. A separate analysis of the 62% missing data for teacher reports showed that older youths with fewer behavior problems at baseline were significantly more likely lack teacher reports. The analysis used the expectation-maximization procedure to impute missing data.

Posttest:

Of the seven outcome measures in Ogden & Halliday-Boykins (2004, Table 1), four showed significant effects. Relative to the control group, the intervention group showed significantly more improvement in internalizing, social competence, out-of-home placement at assessment, and out-of-home placement over the previous six months,

Long-Term:

Using the truncated sample of 75 participants at the 18-month follow-up, Ogden and Hagen (2006) found marginal program effects on out-of-home placements but found significant effects for the subsamples of boys, older youths, and those living at home at baseline (see Tables 1 and 2). Tests for 10 other outcomes (see Table 3) showed six significant effects. Compared to the control group, the MST youths reported lower delinquency than control youths, parents reported fewer behavioral problems and less internalizing, and teachers reported fewer behavioral problems, less internalizing, and less externalizing problems. The effect sizes were 0.26 for self-reported delinquency, 0.50 for parents' ratings, and 0.68 for teachers' ratings.

Quasi-Experimental Comparisons with New Intervention Group (Ogden et al., 2007)

This report used 50 participants from two of the four sites in the original year 1 sample. It also examined 55 new participants at the two sites who were all assigned to the MST condition during year 2 of the program implementation. The design compared the non-randomized MST participants in year 2 to the randomized MST and control participants in year 1. The study lacked randomization in year 2 and made comparisons across two different years. The analysis controlled for the baseline outcomes to adjust for group differences. The results demonstrated sustained effectiveness of the program over time. Compared to the control participants in year 1, the intervention participants in year 2 reported significantly better outcomes for preventing out-of-home placement, youth-reported delinquency and internalizing, and teacher-reported internalizing and externalizing. Further, comparisons of intervention participants showed similar improvements in year 2 as in year 1.

Gender differences in MST outcomes without a Control Group (Ogden and Hagen, 2009)

This report combined data for MST participants from two sample. The first was the RCT (Ogden & Halliday-Boykins, 2004; Ogden &Hagan, 2006) in which 100 adolescents were randomly assigned to either MST (n = 62) or a usual services control group (n = 38). The second was the QED (Ogden et al, 2007) in which an additional group of 55 non-randomized adolescents received MST. Ogden and Hagan (2009) included the 117 youth who received MST treatment in the program's first and second year of operation. The study did not have a control group. No gender differences were found in change from baseline to follow-up for externalizing and attention problem scales, drug use or out of home placement. According to teachers, boys and girls showed similar levels of behavioral change following treatment. Overall, MST was effective regardless of gender.

Study 8

Summary

Leschied and Cunningham (2002) conducted a randomized controlled trial that assigned 411 Canadian youths at high risk for criminal offending to an MST intervention group or a usual-services control group. Assessments from posttest to a three-year follow-up measured family relations and criminal convictions.

The study found that, relative to the control group, the MST intervention group was significantly:

• Less likely to be sentenced to a term of secure custody for three of the four sites
• Improved on parent reports of family adaptability
• Improved on youth externalizing behavior and internalizing symptoms.

Evaluation Methodology

Design:

Recruitment: The sample included 411 youths referred to treatment from four Canadian communities, at high risk for committing a criminal offense, and with problems suitable for MST treatment. Across all four sites, 64% of the youth were referred to the project by probation officers. Although there had to be evidence of past criminal behavior to qualify for the study, about one third had no record of prior criminal convictions at referral.

Assignment: This study randomly assigned the youth and families to the MST group or a usual-services control group. The authors noted that the usual services received by the control group were of higher quality than the services youth receive in the U.S.

Assessments/Attrition: Assessments occurred at baseline and posttest (i.e., after the last session for the MST group and six months after consent for the control group). Data from youths and parents was complete at baseline, but only 57% of the teachers provided baseline data. At posttest, about 62% of the youths and 35% of the teachers provided data. If an MST case closed prematurely, post-testing was not administered.

In addition, all youths were tracked for offending at 6, 12, 24, and 36 months after the case was closed. After discharge, 92% were tracked through six months, 78% through one year, 47% through two years, and 20% through three years.

Sample:

The average age across the four sites ranged from 13.9 to 15.3, with an overall average of 14.6 years. The sample included 26% females and 13% self-identified aboriginals. About one-third were welfare dependent with poor educational achievement and low SES. Also, about 30% of the sample was middle class families with good educations and high incomes.

Measures:

The outcome measures came from multiple instruments: (1) Standard Client Information System (SCIS); (2) Beliefs and Attitudes Scale; (3) Family Adaptability and Cohesion Scale - II (FACES; (4) Social Skills Rating System; (5) Parental Supervision Index. Follow-up data on recidivism and correctional service utilization came from the Canadian Police Information Centre.

Analysis:

The analyses compared group means for the survey outcomes and estimated survival models for conviction outcome.

Missing Data Method: The analysis appeared to use complete cases, without imputation or FIML

Intent to Treat: The study dropped participants (19%) who did not complete MST and thus appeared to use a non-intent-to-treat sample.

Outcomes

Implementation Fidelity:

The Canadian MST program was implemented with a lower budget and less supervision than previous MST studies. This may have undermined the fidelity of program implementation.

Baseline Equivalence:

Tests showed several differences between conditions.

Differential Attrition:

Attrition rates by condition differed substantially. Posttest data were unavailable for either a parent or a youth in 49% of the control group and 28% of the MST group. The members of the MST group had a significantly higher response rate at discharge (71%), compared with the members of the usual services group (52%). Analyses comparing MST dropouts and MST completers show that MST dropouts performed poorly compared with both the MST completers and the usual services group. The researchers were unable to identify the usual services dropouts.

For the teacher data, response rates were very low but similar for the two conditions. Among the 57% of cases for which the teacher information was available, the members of the control group had higher levels of externalizing behavior problems at pretest.

Posttest and Long-Term:

There were no significant mean differences between conditions in rates of convictions, and the survival curve did not differ significantly. In three of the sites, MST participants were significantly more likely to be sentenced to a term of open custody and significantly less likely to be sentenced to a term of secure custody than control participants. There were no significant differences between groups for the length of time youths spent in custody or for the total number of offenses for which youth were prosecuted.

MST youth improved significantly on some problems as measured by psychological testing. Compared to the control group, the MST youth improved significantly on parent reports of family adaptability, caregiver depression, and youths' externalizing behavior. The MST group also improved significantly on youth report of internalizing symptoms when compared to the control group. While there were no significant differences between groups regarding parental supervision.

Study 9

Summary

Weiss et al. (2013, 2015) conducted a randomized controlled trial that assigned 164 youths enrolled in behavior intervention classrooms to an MST intervention group or a no additional treatment control group. Assessments from posttest to a one-year follow-up measured conduct problems, externalizing, and criminal offending.

The study found that, relative to the control group, the MST intervention group showed significantly lower:

• Parent- and adolescent-reported externalizing problems
• Absenteeism in school.

Evaluation Methodology

Design:

Recruitment: The authors contacted public junior and high schools to recruit adolescents between the ages of 11 and 18 who were enrolled in self-contained behavior intervention classrooms that served as restrictive settings designed for adolescents with conduct problems. Of 213 families with children in classrooms, 164 (77%) provided consent to participate in the study. The sample scored highly on baseline measures of externalizing and internalizing.

Assignment: After baseline measures were collected, 84 adolescents were randomly assigned to the treatment group and 80 were randomly assigned to a treatment-as-usual control group that did not receive the treatment.

Assessments/Attrition: Outcomes were assessed at baseline, three months (mid-program), six months (posttest), and 18 months (one-year follow-up). By the one-year follow-up, 7% of the families withdrew from the study or could not be located.

Sample:

Participants were enrolled in the 7th through 11th grades at baseline, with a mean age of 14.6 years. The vast majority of adolescents (83%) were male, 60% were black, and 40% were white. Parents or guardians averaged 40.8 years of age, 71% had at least completed high school, and the median reported family income was $17,500.

Measures:

Weiss et al. (2013): All measures were obtained from separate interviews for parents and adolescents at a location of their choosing, while teachers were given the assessment materials to complete on their own time. Parents both participated in the program to help their child and provided many outcome measures. However, an additional analysis found that adjusting for a measure of positive impression among parents did not change the results.

The primary outcomes were adolescent conduct problems as assessed by parent, adolescent, and teacher reports on the Child Behavior Checklist (CBCL) and criminal charges obtained from court records. The CBCL is used to obtain a broad measure of child behavioral and emotional problems, and its 118 items produce separate scales for externalizing and internalizing problems. The authors report that these scales have average 1-week test-retest reliability of .89. Criminal records were obtained with parental and child assent from juvenile court records from 1 year prior to baseline through 2.5 years post-baseline. Charges were coded as status offenses, misdemeanors, or felonies, with the analysis focusing on incidents wherein the adolescent was charged with a felony.

Secondary outcomes included several areas of conduct problems such as delinquency and drug use that were measured using the Self Report Delinquency Scale (validity and reliability were not reported). School functioning data was also included, and consisted of the student's average grade across all core academic subjects, school attendance, and the number of days suspended during the study period.

The authors also detailed the measurement of nine factors related to the study's risk and protective factors. To assess family relationships, parents and youth completed the Family Adaptability and Cohesion Evaluation Scales III; to assess authoritarian, authoritative, and permissive parenting behavior, the adolescent's primary caregiver completed the Parental Authority Questionnaire; and to assess parent mental health problems, parents completed the measures of externalizing and internalizing from the Personality Assessment Inventory.

Weiss et al. (2015): Three externalizing scales from the parent-reported Child Behavior Checklist, the Youth Self-Report form, and the Teacher Report Form served as the outcomes (alpha values were .89, .85, and .93, respectively).

Analysis:

The analysis used hierarchical linear models for repeated measures with time nested within persons. The models specified time as a random effect with linear and quadratic terms. These models inherently adjusted for baseline outcomes. \

Cox proportional hazards models were used to analyze the court data, and estimated whether time to first felony arrest differed by treatment group, adjusting for whether the adolescent had been arrested in the previous year.

Missing Data Method: The analysis used FIML to include participants with missing data.

Intent-to-Treat: The study adhered to the principles of intent-to-treat, using all available data for all respondents, regardless of program completion.

Outcomes:

Implementation Fidelity:

Quantitative measures indicated that therapists' overall adherence to Multisystemic Therapy principles was moderately high to high, and consistent with the other studies. With the average treatment lasting 5.19 months, 88% of cases were considered to have been closed successfully, and 8% were considered to have been closed partially successfully.

Baseline Equivalence:

Of 15 demographic and baseline primary outcome measures, one differed significantly between treatment groups. The treatment group had a higher proportion of parents who had graduated from high school. The authors state that this variable was not related to treatment outcomes, so it was not included in any of the outcome analyses. Also, the study did not appear to test for differences in the other 14 secondary and risk/protective outcomes.

Differential Attrition:

Over the course of the study, 3 families in the treatment group (3.6%) and 8 families (10%) in the control group withdrew from the study or could not be located by the 1-year follow-up assessment. The authors did not assess differential attrition but used FIML to adjust for missing data in the analysis. Also, the level of overall attrition rate combined with the condition difference in the attrition rates shows little potential for bias according to the WWC cautious and optimistic standards.

Posttest and Long-Term:

In Weiss et al. (2013), two of four primary outcome measures were improved relative to the control group, with intervention-group parents and adolescents reporting greater decreases in externalizing problems; however, teacher reports and arrest data did not show significant treatment effects. Only one of five secondary outcomes differed significantly between groups, with a quadratic treatment effect for the number of days absent in favor of the treatment group. Among nine risk and protective factors, two showed significant improvement in the treatment group: decreased permissive parenting behavior and decreased parent internalizing mental health problems.

Weiss et al. (2015) briefly summarized the main effect results, with significantly greater declines for the treatment than the control group for parent-reported externalizing and youth-reported externalizing but not for teacher-reported externalizing. The major part of the analysis examined moderation of intervention effects by parenting measures. The results showed that families sometimes gained from MST when parents reported maladaptive processes (e.g., low baseline levels of adaptive child discipline skills) but more often gained from MST when parents reported strengths that facilitated the treatment (e.g., high levels of adaptive functioning).

Study 10

Summary

Fain et al. (2014) used a quasi-experimental design that examined 1,137 juvenile offenders from Los Angeles County who were assigned to MST treatment. Those participating in the treatment served as the intervention group and those not participating served as the control group. Assessments after six months measured arrests and incarcerations.

The study found no overall beneficial effects of MST relative to the comparison group but showed improvements in:

• Rates of rearrest, incarceration, and completion of community service among Hispanic youth only.

Evaluation Methodology

Design:

Recruitment: The study sample included 1,137 juvenile offenders who qualified for MST participation based on the Los Angeles County eligibility criteria.

Assignment: The program was evaluated using a QED without randomization. The intervention group consisted of 757 individuals who were accepted into and participated in Los Angeles' MST program between January 2003 and December 2010, and the comparison group contained 380 youth who were eligible but did not participate in MST between January 2001 and December 2010, most often because of lack of Medicaid coverage. There was little information on the kind of treatment that the comparison group received. The study did not employ matching procedures, but the authors noted "To maximize compatibility between MST and comparison youth, we excluded youth from the comparison group if they were unreceptive to program services at the intake session, as well as those whose families were not receptive to MST services."

Assessments/Attrition: The participants were tracked for six months after qualifying for the program (in the comparison group) or six months after entering the program (in the intervention group). The full sample (100%) had arrest and incarceration data, with fewer having data on completing probation (95%), probation violations (94%), completing restitution (65%), and completing community service (44%). Risk and protective factors targeted by the intervention were only collected from the treatment group beginning in 2004, resulting in 508 participants (67% of treatment group) with data in this area.

Sample:

The mean age of program participants was 15.3 at baseline, and almost all youth (96%) had at least one prior arrest. The majority of participating youth were male (77%) and Hispanic/Latino (about 74%), with a smaller number identifying as African American (about 20%) and very few as White or other ethnicity (about 7%). Most participants were arrested for violent (30%) or other crimes (43%) that led to their qualifying for MST, with fewer arrested for property (21%) or drug crimes (5%).

Measures:

Juvenile justice outcomes, all measured as zero for no and one for yes, included 1) arrests, 2) incarcerations, 3) successful completion of probation, 4) successful completion of restitution, 5) successful completion of community service, and 6) probation violations. All were measured over a six-month period and were obtained from an automated database maintained by the Los Angeles County Probation Department.

Five risk and protective factors targeted by the program were measured among the treatment group at baseline and discharge from the program, beginning in 2004. The factors included parenting skills, family relations, network of social supports, success in educational or vocational settings, and involvement with prosocial peers. Performance in each area was rated as satisfactory or unsatisfactory according to criteria specified by MST caseworkers; for example, improvement in parenting skill required parents to demonstrate at least two of the following: 1) increased limit setting, 2) established and enforced consequences, 3) increased monitoring. MST caseworkers who delivered the therapy provided the scores.

Analysis:

The analysis used logistic regression with controls for race/ethnicity, age, gender, and type of offense at recent arrest. McNemar's test (paired chi-square) was used to evaluate the significance of pre- to post- changes in risk and protective factors and did not include adjustments for sociodemographic or criminogenic factors.

Missing Data Method: The analysis used participants with complete data, without imputation or FIML.

Intent to Treat: The analysis included all participants with complete data in their original condition.

Outcomes:

Implementation Fidelity:

No measure of implementation fidelity was reported.

Baseline Equivalence:

The treatment and comparison groups were similar at baseline, significantly differing only on ethnicity (MST participants were more likely to be Hispanic). While baseline equivalence was not examined for the study outcomes, similar measures such as having a prior arrest and offense type at last arrest did not differ by group. However, the groups likely differed on unmeasured socioeconomic characteristics that partly determined assignment.

Differential Attrition:

While the study did not examine differential attrition, all subjects were retained for the analysis of arrests and incarcerations. The smaller sample sizes for other outcomes are likely a result of variation in the sentences given to participants at trial.

Posttest:

After controlling for race, age, gender, and type of offense, the treatment group showed significant improvement in 3 of 6 outcomes (arrests, incarcerations, and completion of community service) relative to the comparison group. Without the controls, the intervention group had significantly more probation violations than the comparison group.

Race-specific analysis of these outcomes revealed that the program's positive effects were observed for Hispanic youth only. While Hispanic MST youth had significantly lower rates of arrest, lower rates of incarceration, and greater odds of completing probation than Hispanic comparison youth, African American MST youth had higher rates of arrest than the comparison group and did not differ significantly on any other outcomes.

Of the 6 functional areas targeted by the program, all (parenting skills, family relations, network of social supports, educational/vocational success, involvement with prosocial peers) were significantly improved at posttest within the treatment group (though with no comparison to the non-MST group).

Race-stratified analyses were also performed, comparing the effects of MST treatment for African American and Hispanic participants.

Study 11

Summary

The study used a randomized controlled trial that assigned 256 Dutch adolescents with antisocial behavior to an MST intervention group or a treatment-as-usual control group. Assessments from posttest to a three-year follow-up measured a variety of outcomes relating to antisocial behavior.

The study found that, relative to the control group, the MST intervention group showed significant reductions in:

• Externalizing
• Oppositional defiant disorder
• Conduct disorder
• Property offenses

Evaluation Methodology

Design:

Recruitment: Adolescents with various types of serious antisocial behavior were referred to the study from multiple community sources across the Netherlands and during the years 2006 to 2010. Of 318 assessed for eligibility, 256 met inclusion criteria and consented to participate.

Assignment: The study randomly assigned the 256 participants to MST (N = 147) or a treatment-as-usual control group (N = 109). The randomization was adjusted using a 1:2 ratio in favor of MST for a six-month period due to a low number of referrals.

Assessments/Attrition: Assessments took place at baseline before the start of the program, about 6 months later at posttest, and at 1.08 years after pretest (roughly 6 months postintervention). Official recidivism data were collected at 3.06 years (mean length, called a 2-year follow-up in the paper) after pretest. They also occurred monthly during the intervention period. About 13% of the participants were lost at posttest and 23% were lost at follow-up.

Sample:

Most of the subjects had a Dutch background (55%). Many belonged to ethnic minorities (34% Moroccan and 32% Surinamese). Half lived in single-parent homes, and 50% of the mothers and 36% of the fathers were unemployed. More than half the families lived below minimum income levels. Judicial records showed that 71% had been arrested at least once.

Measures:

Baseline and posttest assessments took place in the homes of the subjects, and "the majority" of research assistants doing the assessments were blind to the assigned condition. Measures came from parent reports, adolescent reports, and observations and showed acceptable reliability. A brief list summarizes the numerous measures:

• Cognitions based on parent-reported sense of competence and adolescent-reported self-esteem, personal failure, and hostility
• Parenting based on parent-, adolescent-, and observer-reported positive discipline, inept discipline, and relationship quality
• Deviant peers and prosocial peers reported by adolescents
• Externalizing behaviors from the parent-reported Child Behavior Checklist
• Oppositional defiant disorder and conduct disorder from the parent-reported Defiant Behavior Disorders rating scales
• Externalizing behaviors from the adolescent-reported Youth Self Report
• Violent offending and property offending from the adolescent-reported Self-Report Delinquency scale
• Official recidivism data from the Judicial Registration System, Dutch Ministry of Justice

Assessments completed within the intervention period were done by phone with an abbreviated set of items for each scale.

Analysis:

Using an intent-to-treat strategy, the study employed three different statistical approaches. First, the analysis examined posttest outcomes for the groups with ANOVA and controls for baseline outcomes. Second, the analysis examined within-intervention trajectories using latent growth models. Third, structural equation models tested for mediation effects.

Missing Data Method: An email to Blueprints from the lead author noted that the expectation maximum likelihood algorithm in LISREL replaced missing data for sociodemographics and relevant baseline outcomes.

Intent to Treat: Given evidence of data missing completely at random, the analysis used all cases with the expectation maximum likelihood algorithm in LISREL.

Outcomes

Implementation Fidelity:

A Therapist Adherence Measure that the investigators collected monthly from the parents assessed adherence to the nine principles of MST. The mean of 4.36 on a scale from 1 to 5 showed acceptable adherence.

Baseline Equivalence:

No significant baseline differences were found on any of the demographic or outcome variables.

Differential Attrition:

Participants lost to postintervention assessment did not differ significantly on any of the assessed variables from those that remained in the study, based on the the additive test for completers vs. attritors (Asscher et al., 2013; 2014; Dekovic et al., 2012). The authors stated that Multiple Imputation was carried out by the expectation maximum likelihood algorithm. Additionally, Dekovic et al. (2012) used the Little's test to provide evidence that data were missing completely at random.

Posttest:

Deković et al. (2012) presented results on 1) intervention effects at posttest, 2) within-intervention change, and 3) mediated effects. First, the program significantly improved four of the five posttest measures compared to the control group: parental sense of competence, positive discipline, relationship quality, and externalizing problems. Only inept discipline did not improve. Second, as shown by differences across groups in the mean slopes, the program significantly increased the within-intervention rates of growth in the same four outcomes as above. Effect sizes similarly increased over time. By the end of the sixth month, effect sizes for the four significant outcomes ranged from about .2 to .5. Third, mediation models showed that participation in MST was significantly related to a greater increase in maternal sense of competence, which in turn predicted an increase in positive discipline. These changes then predicted a decrease in adolescent externalizing. However, a similar model using relationship quality rather than positive discipline did not significantly mediate between the program and externalizing behavior.

Asscher et al. (2013) found significant improvement in the treatment group relative to the control group for 5 of 6 primary measures: parent-reported externalizing, opposition defiant disorder, and conduct problems, and adolescent-reported externalizing and property offending. Effect sizes ranged from .25-.36. They also found significant program improvement for 9 of 15 secondary measures. The significant outcomes included measures of parent-reported sense of competency, youth-reported hostility, parent-, adolescent-, and observed-reported positive parenting, parent- and observer-reported quality of relationship, observer-reported inept discipline, and adolescent-reported prosocial peers. However, one significant iatrogenic effect emerged for adolescent-reported sense of personal failure. Effect sizes for the secondary measures ranged from .26-.47.

Asscher et al. (2013) also tested for moderation effects by ethnicity and age, with little evidence of differences across groups. Tests for moderation by gender found that for adolescent cognitions, the treatment had larger (and more positive) effects for boys than for girls.

Manders et al. (2013) tested for additional moderation effects. They found that MST was more effective than the control in decreasing externalizing problems for the "lower callous/unemotional" and "lower narcissism" group, but not for the "high callous/unemotional" and "high narcissism" group.

Long-term:

Main Effects (Asscher et al., 2014): There were no differences between the conditions in frequency or number of arrests, time to re-arrest, or type of arrest, at the six-month and 2-year follow-ups. At the end of the final follow-up period, 63% of the MST group and 53% of the TAU group had been rearrested at least once, but these differences were not significant.

Moderation (Asscher et al., 2018): The study examined moderation tests for differences in program effects for extremely violent and not extremely violent youth. The results showed no significant moderation at posttest for externalizing or relationship quality. However, additional analyses of monthly data from pretest to posttest demonstrated different patterns of change. The extremely violent youth worsened initially but then showed large improvements in both externalizing and relationship quality.

Study 12

Summary

Timmons-Mitchell et al. (2006) conducted a randomized controlled trial that assigned 93 juvenile offenders to an MST intervention group or a treatment-as-usual control group. Assessments through 18-month follow-up measured reoffending and youth functioning in eight domains.

The study found that, relative to the control group, the MST intervention group showed significantly:

• Lower rates of reoffending and rearrest
• Greater improvements in four areas of functioning (home, school, community, and moods and emotions)

Evaluation Methodology

Design:

Recruitment: Participants were 93 youth who appeared before a family county court in a Midwestern State between October 1998 and April 2001. Youth were recruited for participation if they met the following inclusion criteria: (a) a felony conviction, (b) a suspended commitment to the Department of Youth Services incarcerating facility, and (c) parents' consent to participate. All youth were either on probation at the time of the study or had been on probation previously.

Parents or legal guardians of youth meeting the study inclusion criteria were asked to consent to random assignment to either MST or TAU. The court agreed to randomize (into the MST or TAU conditions) families in which caregivers and youths agreed to participate in the study. Both parents and youth provided informed consent, which was obtained either by court personnel or by the MST supervisor. Overall, 89% of eligible participants who met inclusion criteria agreed to participate in the study (n=105). If parents or guardians did not agree but were eligible for MST, it was at the discretion of the court whether to assign them to MST without study participation.

Assignment: Randomization was accomplished by having the court administrator flip a coin. 48 youths were randomly assigned to the MST condition and 45 youth were assigned to the TAU condition.

Assessments/Attrition: 89% of participants completed the study (i.e., 11% of study participants dropped out resulting in a final analysis sample of 93 participants). Recidivism was tracked through an 18-month post-treatment follow-up.

Sample:

The mean age of all youths was 14.1 years at the time of enrollment in the study. Twenty-two percent of the participants were female, and seventy-eight percent were male. The racial composition of the sample was as follows: 15.5% African American, 77.5% European-American, 4.2% American Hispanic and 2.8% biracial.

Measures:

Official charge data: The county family court keeps detailed information regarding juvenile arrests. The recidivism analyses in this study were based on those charges for which the youth was formally arraigned following discharge from treatment (for the MST group) or at 6 months post-recruitment (for the TAU group). Charge data were examined through 24-month post-recruitment for both groups. Despite the level of detail in the court record concerning rearrests, few details were available on the specific type of new charge; however, each charge was designated as either a misdemeanor or a felony.

CAFAS: Ratings for the Child and Adolescent Functional Assessment Scale (CAFAS) were made corresponding with the beginning of treatment, discharge, and 6 months following discharge for MST youth. For TAU youth, ratings approximated timing of administration for MST youth (i.e., baseline, 6-month post-recruitment, and 12-month post-recruitment). The CAFAS measures youth functioning in eight areas: school and work, home, community, behavior, substance use, and thinking. For each subscale, the child receives a score of 0, 10, 20, or 30. A score of zero indicates no or minimal impairment, 10 indicates mild impairment, 20 indicates moderate impairment, and 30 indicates severe impairment. A total score based on the summation of the subscales reflects overall youth functioning. The range of possible scores on the CAFAS is 0 to 240. The higher the CAPAS score, the greater the functional impairment.

Brief subscale descriptions provided in the current edition of the CAFAS manual are as follows: (a) School/Work: ability to function satisfactorily in a group education environment, (b) Home: extent to which youth observes reasonable rules and performs age appropriate tasks, (c) Community: respect for the rights of others and their property and conformity to laws, (d) Behavior Toward Others: appropriateness of youth's daily behavior, (e) Moods/Emotions: modulation of the youth's emotional life, (f), Self-Harmful Behavior: extent to which the youth can cope without resorting to harmful behavior or verbalizations, (g) Substance Use: youth's substance use and the extent to which it is disruptive, and (h) Thinking: ability of youth to use rational thought processes.

To provide a standardized method of CAFAS assessment, research assistants used court records to rate MST and TAU youth at baseline, at immediate posttreatment (for MST youth) or 6-month post-recruitment (for TAU), and at 6-month post-treatment (for MST youth) or 12-month post-recruitment (for TAU youth).

Analysis:

A 2 (treatment condition: MST vs. TAU) x 3 (time: pretreatment, immediate post-treatment, and 6-month post-treatment follow-up) mixed factorial design was used to evaluate functioning. Relative odds ratios were calculated for the likelihood of rearrest. Binary logistic regression was conducted to compute the relative risk of rearrest in the TAU versus the MST group. The survival analysis was conducted using a Fleming-Harrington test that weighted offenses later in time more heavily than offenses earlier in time. The rationale for using this weighting scheme with post-treatment arrest data derives primarily from knowing that rates of reoffending typically increase over time as active supervision of youths' activities decreases.

General linear modeling repeated measures were used to examine changes in six CAFAS subscale scores over time. The CAFAS total score is not presented as there is limited evidence that this score presents information separate from that already covered in the subscales. To minimize the effect of multiple tests made on the data, alpha was set at .008. Using the adjusted alpha, time by interaction effects were tested.

Missing Data Method: The study used complete case analysis without imputation or FIML estimation.

Intent-to-Treat: The study referred to including participants who "completed the treatment and completed the study." The wording implies that only program completers were kept for the analysis.

Outcomes

Implementation Fidelity:

Not examined.

Baseline Equivalence:

There were no statistically significant between-group differences with respect to race or sex. There were also no statistically significant between-group differences on court-related variables such as age at first offense, number of pre-treatment offenses, number of pre-treatment misdemeanors, and number of pre-treatment felonies. However, these results appeared to use the analysis sample rather than the randomized sample. Further, tests using the analysis sample in Table 2 show several significant differences between the conditions at baseline (i.e., time 1).

Differential Attrition:

The study did not refer to tests for attrition or show condition differences in attrition rates, but the tests for baseline equivalence of conditions using the analysis offer evidence of differential attrition (see Table 2 for time 1).

Posttest and Long-term:

At the 18-month post-treatment follow-up, the recidivism rate for the MST group (66.7%) was significantly lower than the overall recidivism rate for the TAU group (86.7%). Youth in the MST group were also arrested and arraigned for significantly fewer offenses. There were no significant between-group differences in the percentage of felonies versus misdemeanors. Youths in the TAU group were 3.2 times more likely than youths in the MST group to be rearrested. For youth with at least one rearrest, the average time to first arrest was 135 days for youths in the MST group and 117 days for youths in the TAU group, a nonsignificant difference.

With regards to youth functioning, both groups evidenced improvement in functioning over time, with the MST CAFAS scores significantly better on four of six subscales: home, school, community, and moods and emotions. The difference between the groups was nonsignificant for substance use and behavior towards others.

Study 13

Summary

Butler et al. (2011) conducted a randomized controlled trial that assigned 108 juvenile offenders in London, England, to an MST intervention group or a treatment-as-usual control group. Assessments from posttest to an 18-month follow-up measured reoffending and problem behavior.

The study found that, relative to the control group, the MST intervention group showed significant decreases in:

• Aggression, delinquency, and psychopathic traits at posttest
• Total offenses and nonviolent offenses at the 18-month follow-up assessment.

Evaluation Methodology

Design:

Recruitment: The study recruited participants from two local youth offending services in North London. Eligible participants were between 13 and 17 years of age, living with a parent or principal caretaker, and under a court referral order for treatment, supervision, or parole. Youth were excluded if they were a sex offender; presented only with substance misuse; were diagnosed with a psychotic illness; or posed a risk to research personnel. Of the 478 referred adolescents, 108 meet the eligibility requirements and consented to participate.

Assignment: The 108 participants were randomized into the MST treatment group (n = 56) or a Youth Offending Teams control group (n = 52). The control group services were extensive and multi-component but, unlike MST, were not delivered in a family context or based on an overarching model or set of principles

Assessments/Attrition: Assessments for the primary outcomes occurred six months before randomization, for the six months covering the intervention period, and then every six months until 12 months after posttest (or 18 months after baseline). The self-report secondary outcome measures were obtained only at baseline and at posttest, after the MST treatment was completed (six months after randomization). Attrition ranged from 1-4% across the outcome measures.

Sample:

The majority of participants were male (82%) and were, on average, 15 years of age. The sample was racially diverse with 34% white, 32% black, 5% Asian, and 24% classified as Mixed/Other. Participants had an average of more than two offenses at intake with more than half the convictions constituting violent offenses. Of the parents, 31% had left school with no academic qualifications; 40% had no vocational qualifications; and 54% were without income. In sum, almost all subjects lived in socioeconomically disadvantaged families.

Measures:

The primary outcome measures came from archival records. They included the number of records of total, violent, and non-violent offending behavior (count data) in the National Young Offender Information System database. In addition, dichotomous variables measured the occurrence of six-month periods free of any offending behavior.

The secondary outcome measures came from self- and parent-rated symptoms of antisocial behavior, delinquency-linked cognitions, personality functioning, and parenting variables. The measures were borrowed from prior questionnaires for which validity and reliability had been demonstrated, but alpha values of the employed scales were not reported.

Analysis:

The analysis employed multilevel models to account for the hierarchical data structure of measurement points nested within persons. Both intercepts and slopes of time were allowed to vary randomly across individuals. Depending on the investigated outcome measure, linear, logistic, or Poisson models were used. All models implicitly control for baseline characteristics.

Missing Data Method: The analysis used FIML estimation to include all participants.

Intention-to-treat: The analysis included all participants in their originally assigned condition.

Outcomes

Implementation fidelity:

Other than to note that control youths attended a significantly greater number of appointments than the MST youths, the study did not provide measures of fidelity.

Baseline Equivalence:

The intervention and control group did not differ significantly at baseline on any of the 12 baseline sociodemographic, offending, or sentencing variables.

Differential attrition:

No tests for differential attrition were performed. Attrition was low and the analysis included all participants with FIML.

Post-test:

For the 21 tests for secondary outcomes measured at posttest, five (24%) group-by-time interaction terms were significant and indicated greater improvement for the intervention group. Significant program effects were observed for parent-reported aggression, parent-reported delinquency, youth-reported delinquency, parent-reported psychopathic traits, and parent-reported positive parenting. The authors reported effect size for changes within conditions but not for condition differences in changes.

Long-term:

For the primary offending outcomes assessed over the period from baseline to long-term follow-up, the group-by-time interaction terms indicated a significantly greater decrease in the proportion with offenses and the count of offenses for youths in the MST group compared to youths in the control group. Differences were also significant for non-violent offenses but not violent offenses (see Table 2). The conditions tended to differ more at longer follow-up than earlier.

Mediating effects: A mediator analysis demonstrated that positive parenting did not account for program-related changes in offending behavior among youths. Similarly, adherence to MST standards (based on parent's independent reports about their therapy) did not mediate the group differences on the primary outcome measure of offense frequency.

Study 14

Summary

The study used a randomized controlled trial that assigned 156 Swedish youths with a diagnosed conduct disorder to an MST intervention group or a treatment-as-usual control group. Assessments at posttest and two-year follow-up measured delinquency, alcohol and drug use, and school attendance.

The study found no effects of MST on parenting, mental health, delinquency, social competence, alcohol use, or drug use.

Evaluation Methodology

Using a clinical sample of youths diagnosed with conduct disorder, Sundell et al. (2008) examined the seven-month posttest results, while Lofhölm et al. (2009) examined the 17-month follow-up results.

Design:

Recruitment: The sample consisted of youths ages 12-17 years who met the criteria for a clinical diagnosis of conduct disorder. Exclusion criteria were (a) on-going treatment by another provider; (b) substance abuse without other antisocial behavior; (c) sexual offending; (d) autism, acute psychosis, or imminent risk of suicide; and (e) the presence of the youth in the home constituting a serious risk to the youth or the family. All youths were referred to the study from March 2004 through February 2005 by child welfare services in 27 local authorities (municipalities) from Sweden's three largest cities and one town. A total of 168 families were asked to participate in the study, and 156 (93%) accepted.

Assignment: Randomization of youths to the intervention group (n = 79) or control group (n = 77) immediately followed initial data collection, when research staff opened a sealed and numbered envelope that contained the result of computer-generated randomization. The randomization used the six MST sites as a blocking variable. The control group youths received treatment as usual, which consisted of social services such as individual counseling, family therapy, mentorship, out-of-home care, and residential care.

Assessments/Attrition: The posttest came about seven months after randomization, and a follow-up came 24 months after randomization or about 17 months after posttest. At the posttest, 14 of the 156 families lacked data (9%). Seven had withdrawn from the evaluation, and seven refused to complete the questionnaire. For specific posttest measures, missing data varied between 9-12% for youths and between 8-12% for guardians, with the exception of the guardian's mental health, where missing data reached 19%. At the follow-up, 16 young people and 15 parents did not complete the questionnaire. For specific measures, missing data varied between 10-11% for young people and between 10-16% for guardians.

Sample:

The sample consisted of 61% boys and 39% girls with a mean age of 15.0 years. At study intake, 67% of the youths had been arrested at least once and 32% had been placed outside of the home at some point during the six months before the study intake. Almost half (47%) of the families spoke a language other than Swedish in the home. A majority of the youths (67%) lived in a single-parent home. Of the mothers, 18% had a college education and 51% were unemployed. Of the families, 61% lived entirely or in part on social welfare grants.

Measures:

Youth Psychiatric Symptoms. Six measures from both caregiver and child ratings of total, internalizing, and externalizing symptoms came from the Child Behavior Checklist and Youth Self-Report. Internal consistency was high (.88-.96).

Sense of Coherence. One youth-reported measure from the Sense of Coherence scale reflected the youth rating of life as comprehensible, manageable, and meaningful. Internal consistency was acceptable (.78).

Self-Report Delinquency. Youths answered 40 questions designed to measure antisocial behavior pertaining to violence, general delinquency, and status offenses. Internal consistency was high (.92). In addition, social workers and parents reported on the number of police reports involving the youth.

Alcohol and drug consumption. Youths reported on the frequency and amount of their consumption of five types of alcoholic beverages (e.g., beer and wine) during the six months before measurement. Youths also reported on their use of six types of drugs (e.g., cannabis and cocaine) during the six months before measurement. The Alcohol Use Disorder Identification Test assessed risky alcohol consumption, dependency, and alcohol-related harm. Internal consistency was high (.88). The Drug Use Disorder Identification Test measured current drug-related problems, excluding alcohol. Internal consistency was high (.93).

Anti-Social Peers. One subscale from the Pittsburgh Youth Study measured relationships with antisocial peers. This measure had moderate internal consistency (.71).

Social Competence with Peers. Youth social competence was measured through caregiver and adolescent ratings of social interactions with peers. Internal consistency was .77 for youths and .84 for guardians.

Social Skills. A scale was adapted from the Social Skills Rating System. Internal consistency was acceptable (.72).

School attendance. Information on school attendance came from the school authorities and was based on the 95% of students who continued on to high school after completing compulsory education at Grade 9.

Out-of-Home Care and Social Services. The two measures came from case-file reviews. Social workers also reported on the type and extent of services received by families before referral.

Parenting Skills. Caregivers as well as youths described parenting skills, with a focus on parental knowledge, parental monitoring, parental soliciting, youth disclosure, and family decision making. Internal consistency was high for youth reports (.88) and parent reports (.89).

Mother's Mental Health. Mothers completed the Symptom Checklist-90 with nine symptom constructs and one global index. Internal consistency was high (.98).

Analysis:

The analyses used repeated measures ANOVA for outcomes with both a pretest and posttest and used one-way ANOVA for outcomes with no baseline measure. The key coefficients came from the treatment-by-time interaction terms.

Intent-to-Treat: The analyses used all cases in their original assignment. Missing data at the posttest were imputed by carrying the baseline measure forward, while missing data at follow-up were imputed with multiple imputation.

Outcomes

Implementation Fidelity:

About 73% of the MST clients completed the program on the basis of the mutual agreement of the primary caregiver(s) and the MST team. This was similar to the average treatment completion rate (74%) for MST programs worldwide. Fidelity scores for 160 interviews with 60 families (alpha = .86) had a mean of 4.00 (range of 1.73 to 5.00), approximately one standard deviation lower than that reported in a U.S. study.

Baseline Equivalence:

Intervention parents exhibited significantly more mental health symptoms than control parents, and control parents reported more children with early-onset behavior problems than intervention parents. No other significant differences existed between the treatment and control groups on demographic or psychosocial variables.

Differential Attrition:

At posttest, the 14 youths who did not complete the questionnaire did not differ significantly from the 142 youths who did on either demographic or psychosocial characteristics. At follow-up, the 16 young people and 15 parents who did not complete the questionnaire did not differ significantly on any of the baseline measures from the 140 young people and 141 parents who did.

Posttest:

Tests reported by Sundell et al. (2008) for 20 outcomes revealed only one significant result: The intervention youth received more days of service than the control youth. Numerous tests for moderation found few subgroup differences in program effects.

Long-Term:

Tests reported by Lofhölm et al. (2009) for 20 outcomes revealed no significant results. Numerous tests for moderation found few subgroup differences in program effects.

Study 15

Summary

The study used a randomized controlled trial that assigned 684 English youths with antisocial behavior to an MST intervention group or a treatment-as-usual control group. Assessments through five years measured out-of-home placement and criminal offending.

The study found several posttest effects of the MST intervention, but the effects disappeared after one year and the program showed no long-term effects.

Evaluation Methodology

Design:

Recruitment: The sample came from nine sites across England. Young people ages 11-17 with moderate-to-severe antisocial behavior were recruited from local service organizations. Those eligible had at least three severity criteria indicating past difficulties across several settings and one of five general inclusion criteria for antisocial behavior: persistent (weekly) and enduring (≥6 months) violent and aggressive interpersonal behavior; at least one conviction plus three additional warnings, reprimands, or convictions; conduct disorder diagnosed according to DSM-IV criteria and not responding to treatment; permanent school exclusion for antisocial behavior; and significant risk of harm to others or self. Between February 4, 2010, and September 1, 2012, 1,076 families of young people were referred as eligible, and 684 families consented to baseline assessment.

Assignment: Eligible families were randomly assigned (1:1) by an independent researcher to the management-as-usual control group (n = 342) or to three to five months of Multisystemic Therapy followed by management as usual (n = 342). The randomization stratified for treatment site, sex, age at study enrolment (<15 years or ≥15 years), and age at onset of antisocial behavior. Management as usual involved the best available local services for young people, as identified by a multi-agency referral panel.

Assessments/Attrition: Outcomes were assessed at baseline and six, 12, 18, 24, 36, 48, and 60 months after randomization. With the program lasting three to five months, the assessments included a posttest and six-month follow-up plus long-term follow-ups ranging from one year to 4.5 years after treatment end. For the primary outcome, which was measured at two time points, 99% of the participants had data at 18 months and 89% at 60 months. For secondary outcomes, which were measured at six time points, 85% of the 684 randomized families remained in the study at six months, 76% at 12 months, 72% at 18 months, 70% at 24 months, 63% at 36 months, and 51% at 48 months.

Sample:

More than 80% of the sample youths met DSM-IV criteria for a conduct disorder (including oppositional defiant disorder). Their mean age was 13.8 years, with 63.5% being male. About 75-78% of the families had low income, and most were white (76-80%).

Measures:

Investigators and research assistants were masked to treatment allocation and were located separately to avoid contamination. The primary outcome in Fonagy et al. (2018) was the proportion of participants in out-of-home placement at 18 months. Secondary outcomes included time to first criminal offense and the number of offenses (total, non-violent, and violent), based on official police records. In addition, Fonagy et al. (2018) examined about 39 other secondary outcomes (see Tables 4-7 for a list). Many relating to conduct problems, delinquency, ADHD, and strengths and difficulties came from questionnaires completed by young people, parents, and teachers. Parents rated their children as well as their own parenting skills and mental health. Because parents and youths participated jointly in the program, parent ratings of the youths can be considered independent. Measures of school attendance and exclusions came from educational records. Psychiatric disorders were assessed by clinicians at baseline and 12 months using the Development and Well-Being Assessment instrument. Appendix Table A.6 shows good reliabilities for nearly all scales.

The primary outcome in Fonagy et al. (2020a, 2020b) was the proportion of young people with any criminal conviction at 18 months and 60 months. The secondary outcomes included the total number of recorded offenses with convictions, time to conviction, and whether the conviction resulted in a custodial sentence, community sentence, or a caution. The numerous self-reported secondary outcomes completed by the young person and their parent or caregiver measured antisocial behavior and attitudes, parenting, family functioning, and the use of healthcare, social care, education, and criminal justice services (see Table 2 in Fonagy et al., 2020b, pp. 10-11). With a few exceptions, the measures had acceptable reliability coefficients (see Appendix 1, Table 20 in Fonagy et al., 2020b, p. 59).

Analysis:

The analysis of out-of-home placement at 18 months in Fonagy et al. (2018) used a mixed-effects logistic regression model, and the analysis of the time-to-event outcomes for first criminal offense used Cox regression. Random effects adjusted for clustering by therapist. The logistic regression model included covariates for site, number of past convictions, sex, and age at onset of criminal behavior. The analysis of secondary outcomes used linear mixed models, with adjustment for baseline values, and Poisson mixed models for count variables.

The analyses in Fonagy et al. (2020a) used mixed-effects logistic regression models for dichotomous outcomes, Cox proportional hazards models for time-to-event outcomes, Poisson mixed models for count outcomes, and linear mixed-effects models for continuous outcomes. All models included the following covariates: prior criminal convictions, sex, age at onset of criminal behavior, site as a random effect, and the baseline outcome. The models used Full Information Maximum Likelihood estimation under the assumption of data missing at random. In addition, the analysis used multiple imputation for all baseline and outcome variables. For the numerous secondary outcomes, the Benjamini-Hochberg multiple testing procedure adjusted for multiple comparisons.

Intent-to-Treat: The analyses included all randomized participants with available data. A sensitivity analysis used multiple imputation but found only minor differences for the results (see the Appendix in Fonagy et al., 2018).

Outcomes

Implementation Fidelity:

The study did not present quantitative analyses, but the authors stated that eight of nine sites performed well above the standards expected by the developers. Detailed cost analyses showed no savings from MST.

Baseline Equivalence:

As shown in Table 1 (Fonagy et al., 2018), the two conditions were similar at baseline, but the study lacked significance tests or d values. The authors noted that slightly more young people with attention deficit hyperactivity disorder were in the intervention group (33%) than in the control group (27%).

Differential Attrition:

At 18 months, the control group had fewer families available for assessment (68%) than the intervention group (75%). Three direct observational points were available for 86% of the intervention families and 81% of the control families. Appendix ii (Fonagy et al., 2018, p. 18) compared the baseline scores of participants who completed the 12-month assessment with those who did not. Although there were no significant differences between these groups at p < .01, there were four significant differences at p < .05 in 42 tests.

Fonagy et al. (2020a) noted that "missing data due to loss to follow-up was not associated with any baseline characteristics, corroborating the representativeness of the sample."

Posttest:

At six months (posttest) or 12 months (six-month follow-up), the program did not significantly affect the number of total crimes, violent crimes, or non-violent crimes.

At six months, Tables 4-6 in Fonagy et al. (2018) show significant program effects for 17 of 33 tests, while at 12 months, the tables show significant program effects for eight of 33 tests. The intervention group did better than the control group on total strength and difficulties (youth reported), emotional problems (youth reported), mood and feelings (youth reported), family satisfaction (parent reported), total strengths and difficulties (parent-reported), emotional problems (parent reported), hyperactivity (parent reported), and parent mental health (parent-reported). In addition, school attendance and exclusion did not differ significantly across conditions, and clinician-rated mental health at 12 months showed no significant effects in six tests (Table 7).

Long-Term:

At 18 months (one-year follow-up), Fonagy et al. (2018) reported no significant program effect on the primary outcome of out-of-home placement or on the secondary outcome of time to first offense (Table 2). At 18 months, there was a significant iatrogenic effect on the mean number of crimes (Table 3). Moderation tests showed three significant subgroup differences in program effects in 19 tests, but all three indicated harmful effects of the program for high-risk youth.

Also at 18 months in Fonagy et al. (2018), Tables 4-6 list significant program effects for two of 33 tests: youth callous and unemotional traits and parent mental health (labeled as General Health Questionnaire in Table 6). By the long-term assessment, none of the earlier significant youth effects persisted, and among the parent risk and protective measures, only self-rated mental health remained significant.

Fonagy et al. (2020a) found no significant differences between conditions at the 60-month follow-up for the primary outcome of a criminal conviction. For the secondary outcomes related to offending, the MST group did significantly worse than the control group at 24 and 48 months, but the difference fell to non-significance by 60 months. For the secondary outcomes at 24, 36, and 48 months, the results showed no condition differences in youth- or parent-reported outcomes after adjusting for multiple tests (see Appendix 3, Tables 22-35 and Appendix 4, Tables 36-45 in Fonagy et al., 2020b).

Study 16

Summary

The study used a quasi-experimental design that examined 320 youths with serious emotional disorders. The participants were non-randomly assigned to an MST-only group, a wraparound-only group, or a wraparound plus MST group that received both interventions. Measures of behavioral problems and global functioning were obtained over an 18-month period.

The study found that, relative to the wraparound-only condition, the MST-only condition showed significantly greater reductions in:

• Parent-rated youth behavioral problems.

Evaluation Methodology

Design:

Recruitment: The sample consisted of 320 children and adolescents ranging in age from 4 to 17.5 years. Participants came from a care center covering a rural and frontier area of Nebraska and targeting youth with serious emotional disorders. Referrals came from schools, welfare agencies, justice organizations, and other child-serving agencies. To qualify for wraparound services, the youths must have demonstrated a diagnosable mental health disorder, functional impairment, and high risk for restrictive placement, dropping out of school, or involvement in the juvenile justice system. Addition criteria for MST services included physical aggression, delinquency, and substance use.

Assignment: In a quasi-experimental design that used no matching, the study examined three groups of youth: a wraparound-only group (n = 213), an MST-only group (n = 54), and a wraparound plus MST group (n = 53) that received both interventions, sometimes at the same time and sometimes in sequence. Assignment appeared to be based on the extent of problems of the youth, as wraparound and MST had some different requirements for treatment and wraparound plus MST was provided to youth who did not respond to their original treatment. The authors noted (p. 159) that "Propensity score matching was also considered to account for baseline differences; however, the sample size was insufficient to allow for reliable estimates."

Assessments/Attrition: Assessments occurred at baseline and three consecutive six-month follow-ups over 18 months. Attrition rates were 11% at six months, 28% at 12 months, and 37% at 18 months. The final analysis sample, which allowed for use of participants without data for all four time points, included 298 participants (93%). Average treatment lengths ranged from 5.5 to 15 months, which means the assessments did not include a long-term follow-up.

Sample:

The sample averaged 12 years of age, and 73% were male. The racial distribution of the sample was 90% white, 4% American Indian, and 6% other. About 11% of the participants were of Hispanic ethnicity. Most families (57%) reported a gross household income of $25,000 or less.

Measures:

The study examined two outcome measures, one reported by parents (total behavioral problems from the Child Behavior Checklist that combined internalizing and externalizing subscales) and one reported by caregivers or clinicians (global functioning across eight life domains from the Child and Adolescent Functional Assessment Scale). Parents participated in but did not deliver the intervention and therefore provided independent assessments of their children. Both instruments are commonly used and well validated, though the authors did not report reliability and validity information for the study sample.

Analysis:

The analysis used repeated measures mixed models with maximum likelihood estimation and an unstructured covariance structure, random coefficients, and random slopes. The models controlled for baseline outcomes, gender, age, minority status, family income, and number of living placements reported at each six-month follow-up. Condition-by-wave interaction terms tested for program effects.

Intent-to-Treat: Models were estimated with imputed and non-imputed data, but because the results did not differ, the study reported results from the non-imputed data.

Outcomes

Implementation Fidelity:

MST-only youth spent 5.5 months in therapy and reported higher participation in family preservation and family therapy than the wraparound-only youths.

Baseline Equivalence:

Table 1 showed eight significant baseline condition differences in 11 tests. The MST-only group was older, included more males, and had fewer problems than the other groups. Youth who received both wraparound and MST had more severe problems than those of the other two groups, likely because they did not respond to the first assigned intervention.

Differential Attrition:

The authors reported that attrition at 18 months did not differ by treatment group, gender, age, minority status, family income, or the baseline outcomes.

Posttest:

The mixed models found, as indicated by the group-by-wave interactions, significantly greater improvements for the MST-only group than the wraparound-only group for one of two outcomes - total behavioral problems from the parent-rated Child Behavior Checklist but not global functioning from the Child and Adolescent Functional Assessment Scale.

Long-Term:

Not examined.

Study 17

Summary

The study used a randomized controlled trial that assigned 674 youths with serious emotional disorders in rural Tennessee to an MST condition or a usual-services condition. Measures of behavioral problems and out-of-home placements were obtained over an 18-month period and used to examine MST effectiveness in counties with and without a countywide service-facilitation program.

The study found that, relative to the usual-services condition, the MST condition showed significantly fewer:

• Out-of-home placements
• Behavioral problems but only in counties with a countywide service-facilitation program

Evaluation Methodology

The study focused on testing the benefits of an organizational intervention called ARC (Availability, Responsiveness, Continuity) that aims to improve the delivery of community-based mental health services. In so doing, the study examined MST alone as well as MST in combination with ARC.

Design:

Recruitment: The sample came from 14 of the poorest, least populated counties in the rural Appalachian region of eastern Tennessee. Between October 2003 and September 2007, youths referred to juvenile court in each of the 14 counties were recruited for the study if they met a list of 11 eligibility criteria, including ages 9-17, referral for a status offense or delinquent behavior, at risk of out-of-home placement, and a diagnosis of serious functioning problems or psychiatric symptoms. A total of 674 youth and their caregivers met the eligibility criteria and agreed to participate.

Assignment: First, randomization at the county level assigned six counties to the ARC intervention and six to no ARC intervention. Second, delinquent youth were randomly assigned within each county to either the MST program or to the usual services program. The second assignment used predetermined, concealed randomization of sequence numbers based on the order of recruitment. Youth assigned to usual services received a variety of inpatient and outpatient mental health services. The double randomization created four conditions that allowed for 1) the comparison of youth receiving MST (n = 349) with youth receiving usual services (n = 325) and 2) the county-level ARC intervention to moderate MST effects.

Assessments/Attrition: The four assessments occurred at baseline, six months, 12 months, and 18 months. Completion rates were 77% at six months, 69% at 12 months, and 52% at 18 months. According to the CONSORT diagram, however, the analysis included 84-88% of the participants who had data for at least two time points. With the MST intervention lasting an average of about 3.5 months, the 18-month assessment defined a long-term follow-up.

Sample:

The youth sample averaged 14.9 years of age, was 69% male, and reflected the racial characteristics of the rural Appalachian region of eastern Tennessee (91% white). Over half (53%) of the youths had two or more mental health diagnoses.

Measures:

The outcomes included two measures. Total youth behavior problems, a combination of internalizing and externalizing, was measured by the caregiver-rated Child Behavior Checklist (alphas ranged between .94 and .95 across waves). Youth out-of-home placements in state custody were measured by interviews with the youth's caregiver. As caregivers participated in but did not deliver the program, the measures appeared to be independent.

Analysis:

The analysis used multilevel random-effects models with youth nested within counties, linear functions for the continuous outcome, and a logit link function for the dichotomous outcome. The equations for the models on pages 545-546 included county as a level-2 factor with random effects that adjusted for within-county clustering. Although not stated explicitly, the models imply the use of FIML estimation.

Missing Data Method: The analysis excluded only those participants with data for less than two time points and used mixed models with FIML to include others with incomplete data.

Intent-to-Treat: The analysis did not include all participants but included many with incomplete data.

Outcomes

Implementation Fidelity:

Detailed analyses compared MST fidelity with and without the ARC. The findings indicated that the inter-organizational components of ARC facilitated greater therapist sensitivity to, and improvement in relations between, the family, community, and service system.

Baseline Equivalence:

The authors reported only that there were no significant differences between ARC and non-ARC county means on total population, per capita income, and percentage of children in poverty. No figures or tests were presented for baseline differences in youth characteristics.

Differential Attrition:

The authors reported that "a pattern-mixture model for the analysis of missing data found no differences in the effects of ARC and MST between youth with and without data at each wave" and that "no differences were found in the effects of ARC and MST between youth with and without data at each wave." Pattern mixture models typically use available data to identify missing data patterns and then include indicators of the patterns as covariates. Also, the use of imputation may moderate potential attrition bias.

Calculations based on the CONSORT diagram indicated attrition of 13-18% in the MST condition versus attrition of 10-14% in the control condition. Based on the overall attrition rate and the difference in attrition rates between conditions, the study met both the WWC cautious and optimistic standards at all assessments.

Posttest and Long-term:

Over the 18-month follow-up period, out-of-home placements were significantly lower for the youths who received MST than for youths who received usual services (OR = .472). ARC did not significantly moderate the effect of MST.

Over the first six months and the next 12 months (18 months total), the measure of behavioral problems did not differ significantly between the MST group and the usual services group. However, moderation tests found that MST was significantly more effective than usual services in ARC countries in the first but not the second period.

Study 18

Summary

The study used a quasi-experimental design that examined 215 youths with serious behavioral problems in Washington State who had been referred to receive MST services. These youths were then matched to comparison youths who had an encounter with the public mental health system. Measures were obtained for criminal convictions and mental health services over one year.

The study found that, relative to a matched comparison group, the MST group showed significantly higher:

• Utilization of mental health services.

Evaluation Methodology

Design:

Recruitment: The study began with adolescents in Thurston and Mason counties of Washington State who met the following criteria: ages 12 to 17 years, exhibited behavioral challenges and significantly interrupted functioning across multiple domains, and faced high risk of being placed out of home. Referrals came mostly from the juvenile justice or mental health systems (71%). Of 219 referrals made from April 2007 through June 2010, 215 enrolled in the study and 183 remained after removing duplicates and those without matches in the administrative database.

Assignment: In this quasi-experimental design, the 183 youths referred to MST served as the intervention group. Comparison youths were ages 9-17 and had an encounter in the public mental health system over the same study period used for the intervention youths. The study identified matched pairs of youths who were similar in age, gender, ethnicity, primary diagnosis, level of functioning, prior mental health service utilization, and criminal history. It was unclear if the comparison youths came from the same counties. For crime outcomes, 101 participants were successfully matched, and for mental health service utilization outcomes, 126 participants were successfully matched.

Assessments/Attrition: The study assessed outcomes over a one-year follow-up period. As it included the treatment time, the one-year assessment period did not include a long-term follow-up. The retrospective study used a sample of youths with existing data, and there was no attrition.

Sample:

The intervention group included 60% males, averaged 14 years of age (with a range from under 10 to 17), and was 83% white, 9% Hispanic, 5% Native American, 2% Asian and Pacific Islander, and 1% African American. About 68% had a misdemeanor or felony conviction in the previous year, and the majority were diagnosed with behavioral and disruptive disorders.

Measures:

Outcome measures came from administrative data gathered by multiple state agencies. The eight measures included criminal convictions (total, misdemeanor, felony, violent) and mental health services utilization (inpatient admission, outpatient treatment, support services, and crises services). The study provided no details on the reliability or validity of the measures.

Analysis:

The analysis used logistic regression with controls for age, gender, race, primary mental health diagnoses, level of functioning, and prior measures for the outcome of interest.

Intent-to-Treat: The analysis used all available data.

Outcomes

Implementation Fidelity:

The study stated that "MST . . . has met adherence standards throughout the project," but also that the program served some youths who did not fit the eligibility criteria.

Baseline Equivalence:

Before matching, MST youths were more likely than comparison youths to be involved in the criminal justice system, use intensive inpatient and other mental health services, be diagnosed with behavioral or attention deficit disorders, and have experienced multiple out-of-home placements. After matching (Exhibit A6.1, p. 7), the conditions still differed on five of 15 baseline measures.

Differential Attrition:

No attrition.

Posttest:

Tests showed no significant condition differences over the one-year follow-up period on any of the four criminal conviction measures (although the percentages favored the MST youth and the effect size for any conviction was -.10). Additional tests for use of mental health services found one significant intervention effect: the MST group had higher utilization rates of mental health support services. However, the author noted that increased use of support services could be characterized as a good outcome (MST promotes access to services) or a negative outcome (MST is associated with increased dependence on public services).

Long-Term:

Not examined.

Study 19

Summary

The study had no control group, instead offering a comparison of MST with Functional Family Therapy. It used a quasi-experimental design that examined 1,256 youths under the care of the Florida Department of Juvenile Justice who were referred to MST or Functional Family Therapy for treatment. The study matched participants in the two interventions using propensity scores (628 in each) and measured criminal offending during the intervention and over the next 12 months.

The study found that, after statistical correction for multiple tests, the MST group did not differ significantly from the matched Functional Family Therapy group on offending during the treatment period or during the 12 months after the treatment.

Baglivio et al. (2014) found no significant effects favoring the MST group compared to the matched Functional Family Therapy group.

Evaluation Methodology

Design:

Recruitment: The sample included all 2,312 juvenile offenders under the care of the Florida Department of Juvenile Justice who were referred to MST or FFT between July 1, 2009, and June 30, 2011. Excluding 109 youth who turned out to not qualify for the programs left 2,203 for the study. The statewide sample came from 10 MST sites and 11 FFT sites within 50 of Florida's 67 counties (both rural and urban for each service).

Assignment: The quasi-experimental design began with 629 youths referred to MST and 1,574 youths referred to FFT. Hundreds of probation officers made the referrals over the two-year period. With MST and FFT rarely available at the same time and the same sites, officers seldom had a choice between the two programs. Officers at all sites used the same form and the same inclusionary/exclusionary criteria for both programs. However, officers in sites with MST may have used different selection criteria for the program than officers in sites with FFT.

The study used propensity score matching to adjust for observed differences across the conditions with nearest-neighbor matching, no replacement, and a caliper of .05 (all analyses were replicated using a more restrictive caliper of .01 and nearest-neighbor matching without replacement). Predictors included gender, race, region of the state, age at admission to service, overall risk to reoffend, criminal history, a global social risk measure, most serious prior offense, age at first offense, antisocial peer association, alcohol use, drug use, and parental authority. Separate matching was done for the full sample, for males, females, whites, non-whites, low-risk youth, and moderate-high to high-risk youth.

The final matched sample included 628 youths in each condition (1,256 total).

Assessments/Attrition: The study examined data for the period of service and for the 12 months after the end of service. Service lasted about four months for MST and about three months for FFT. The retrospective data had no attrition.

Sample:

The matched sample was 69% male and 48% non-white.

Measures:

The two outcome measures came from state administrative records. First, a dichotomous measure of offending during service included any new criminal offense referral (arrest) or violation of probation while receiving or enrolled in MST or FFT. The longer time of service for the MST group could affect the offending measure.

Second, a dichotomous measure of recidivism captured whether the youth was adjudicated or convicted for an offense that occurred within 12 months of termination of MST or FFT (termination of service could be either successful or unsuccessful completion). The measure included data from adult records for youths who turned 18 years of age during the follow-up period. The recidivism measure, unlike the offending measure, used adjudicated/convicted offenses rather than arrests.

Analysis:

The analysis used t-tests with a Bonferroni correction for two outcomes (.05/2 = .025) in comparing the two matched conditions.

Intent-to-Treat: The analysis used all youth referred to the services, even if they did not successfully complete the program.

Outcomes

Implementation Fidelity:

The study noted only that the completion rate for MST of 71.9% was significantly lower than for FFT (76.5%).

Baseline Equivalence:

Many significant condition differences existed before matching, with MST having lower scores on percent male, risk, criminal history, and seriousness of priors. However, after matching, there were no significant differences in tests for 13 matching variables. Effect sizes for the differences were all below .25.

Differential Attrition:

No attrition.

Posttest:

With the usual significance tests, the FFT group had significantly lower offending during services than the MST group, but the effects fell to non-significance with a Bonferroni correction and with matching using a .01 caliper. Some evidence suggests that the significant effects favoring FFT were strongest for young women and low-risk youth.

Long-Term:

Recidivism for the 12 months after the end of treatment did not differ significantly across the two conditions.

Study 20

Summary

The study had no control group, instead offering a comparison of MST with Functional Family Therapy. It used a quasi-experimental design that examined 697 youths from a Dutch institute for behavioral problems who were referred to MST or Functional Family Therapy for treatment. The study matched participants in the two interventions using propensity scores and measured posttest externalizing as the primary outcome.

The study found that the MST group did significantly better than the matched Functional Family Therapy group on:

• Engagement in school or work at posttest.

Evaluation Methodology

Design:

Recruitment: The sample consisted of 1,714 adolescents and their families who started either Functional Family Therapy (FFT) or MST at an institute for personality disorders and behavioral problems in the Netherland between October 2009 and June 2014. About 45% of the youths had a court order for treatment and were considered high risk. Of the 1,714 youths undergoing treatment, 697 completed the treatment and comprised the final sample.

Assignment: Adolescents with more serious problems were assigned to MST (n = 1074) and those with less serious problems to FFT (n = 640). After dropping those not completing the treatment, 422 youths (39%) remained in MST and 275 youths (43%) remained in FFT. To adjust for differences, the study used weights based on propensity scores. The covariates used in the propensity model (Table 1) included measures of sociodemographic characteristics, family relationships, and the baseline outcomes. Use of covariate missing values as predictors allowed for inclusion of all participants in the propensity score model.

The MST weight was 1, while the FFT weight was an odds ratio (i.e., the propensity score divided by one minus the propensity score). The propensity scores were estimated for the full sample and also for the subsamples of youths with and without court orders.

Assessments/Attrition: The posttest assessment occurred at the end of treatment. With data only on participants who finished treatment, 697 (40.7%) completed the primary outcome measures.

Sample:

Of the adolescents who completed MST, about 65% were male and about 81% were born in the Netherlands. They had an average age of 15.7 years.

Measures:

Externalizing as reported by both parents (Child Behavior Checklist) and youths (Youth Self Report) served as the primary outcome, with the measures having alphas of .94 and .88, respectively. The three secondary outcomes included 1) living at home (i.e., no out-of-home placement); 2) engagement in school or work for at least 20 hours per week at the end of treatment; and 3) new police contact due to inappropriate or illegal behavior during the treatment period. The therapists reported the three secondary outcomes as part of a standard scoring procedure that supervisors and consultants monitored.

Analysis:

The analysis used linear regression (with the propensity score weights) for continuous outcomes and logistic regression for binary outcomes. Tests of significance used 5,000 bootstrap samples to obtain non-parametric 95% confidence intervals.

Intent-to-Treat: Using only participants who completed treatment, the study examined a treatment-of-treated sample rather than an ITT sample.

Outcomes

Implementation Fidelity:

The mean adherence score within MST was .53, which was in accordance with the norm of ≥0.50 provided by MST Institute.

Baseline Equivalence:

Before matching, tests showed numerous significant differences and large standardized differences (d > .25) between the conditions (Table 1). After matching, the standardized differences all fell to below .25. Also after matching, the variance of key measures was similar across conditions. However, the authors noted that eight participants from the MST group and 12 participants from the FFT group were removed from the baseline tests because "there was no overlap" in the propensity scores.

Differential Attrition:

Those completing the treatment and having outcome data differed significantly from those who did not with regard to their country of birth, living situation, and whether or not they had a court order before treatment, as well as the level of education and employment status of their primary caregiver, and whether or not this primary caregiver had a partner.

Posttest:

One of five outcomes showed significant condition differences at posttest: A significantly higher proportion of adolescents who had completed MST versus FFT were engaged in school or work after treatment (d = .19). Tests for subgroup effects found that among adolescents without a court order, MST did significantly more than FFT to reduce both parent- and child-reported externalizing problems.

Long-Term:

Not examined.

Study 21

Summary

The study had no control group, instead offering a comparison of MST with an adapted version of MST for adolescents with intellectual disabilities (MST-ID). It used a quasi-experimental design that examined 63 Dutch adolescents with intellectual disabilities who were referred to MST or an adaptation of MST for adolescents with intellectual disabilities. The study matched participants in the two interventions using propensity scores and measured living arrangements, school/work, and police contacts at posttest and six-month follow-up as the primary outcomes.

The study found no significant effects favoring the MST-only group.

Evaluation Methodology

Design:

Recruitment: The sample, obtained in the Netherlands between March 2014 and October 2015, included families of adolescents with antisocial or delinquent behavior and with intellectual disabilities (i.e., an IQ score of 50 to 85). In addition, eligible participants were at least age 12 and were receiving no other treatments. Of the 214 families who met the inclusion criteria, 128 families (60%) consented to participate.

Assignment: Dutch agencies referred families to standard MST (n = 73) or MST‐ID (n = 55), a program that adapted MST for adolescents with intellectual disabilities. The alternate treatment was provided by two teams from one organization specializing in care for people with intellectual disabilities, while the standard MST was offered by 24 teams from seven Dutch organizations.

The study used propensity score weights based on 27 predictors to adjust for condition differences. All observed baseline variables, as well as missing indicators for all baseline variables with missing data, were included as predictors in the propensity score model. The missing data indicators allowed all families to be included in the propensity score estimation. The resulting propensity scores were used as weights equalizing condition differences. However, limiting the matched participants sample to those with overlapping propensity scores reduced the sample size from 128 to 63. The differences for 40 families in the MST group and 25 in the alternative group were too great to include in the analysis (i.e., scores did not fall in the range of scores observed in the other treatment group).

Assessments/Attrition: Assessments occurred at baseline, posttest, and six-month follow-up. Six months after treatment, 87 families (68%) provided data. As noted above, additional losses came from non-matching propensity scores, which reduced the sample to 63 families (49%).

Sample:

Across the two conditions, the percent female in the sample ranged from 36-44%. The average ages were 14.9 and 15.2. About 95% of the sample was born in the Netherlands.

Measures:

The outcome measures came from the reports of therapists and parents at baseline and posttest and from parents at follow-up. The three primary outcomes included if the adolescent was living at home, attending school or working for at least 20 hours a week, and free of police involvement since the start of treatment. At posttest, therapist reports were monitored by the team supervisor and the MST consultant.

The study also examined eight secondary outcomes at posttest relating to parenting, family relations, and youth externalizing. At the follow-up, only the rule-breaking component of externalizing was examined. The study reported no information on reliability or validity for the secondary measures.

Analysis:

The analysis used logistic regression for dichotomous outcomes and linear regression for continuous outcomes. Bootstrapping was used to calculate 90% confidence intervals for all outcome measures, with 5,000 bootstrap samples drawn from the weighted sample.

Intent-to-Treat: The study did not drop participants for not completing the treatment, though many subjects were lost in the matching.

Outcomes

Implementation Fidelity:

On a scale ranging from 1 (not at all) to 5 (very much), the average therapist adherence score was 4.38 for standard MST.

Baseline Equivalence:

Using the full sample of 128, there were five significant condition differences before matching. After matching, significance tests were not reported, but all baseline variables as well as the missing indicators used in the propensity score estimation had standardized mean differences lower than 0.25. This result came from excluding the 65 families with non‐overlapping propensity scores and using the analysis sample of 68.

Differential Attrition:

The study did not test for differences between those with and without posttest and follow-up data.

Posttest:

All significant effects favored the alternative treatment. Based on a 90% confidence interval, none of the three primary outcomes differed significantly at posttest, while one, living at home, favored the alternate treatment at six-month follow-up. Several secondary outcomes were significantly better for the alternate treatment group.

Long-Term:

Not examined.

Study 22

Summary

The study used a quasi-experimental design that examined 740 youths referred to MST by the state of Rhode Island. The intervention group completed the treatment, while the comparison group did not complete the treatment for reasons unrelated to case progress. After using propensity scores to match the two conditions, the study examined delinquency outcomes over the next six years.

The study found that the intervention group relative to the comparison group showed significantly lower risks for

• Out-of-home placements
• Adjudications
• Juvenile training school.

Evaluation Methodology

Design:

Recruitment: The sample came from five MST providers that were active in Rhode Island from 2008-2011. A total of 772 youths ages 11-17 were referred to MST by either Child Welfare Services or Juvenile Correctional Services in Rhode Island, primarily for reasons of behavioral problems, disobedience, truancy, and delinquency. The youth who received an out-of-home placement prior to initial program contact were excluded, leaving a final sample of 740.

Assignment: The treatment condition included all youth referred to MST who either completed treatment or had the opportunity to do so. The latter group included youth who were unable to complete treatment due to either lack of engagement or a placement occurring while involved in treatment. Of the total sample of 740, 78% (n = 577) met criteria for inclusion in the treatment condition, with 90% (n = 522) completing treatment, 8% (n = 46) discharged due to placement, and 2% (n = 9) discharged due to lack of engagement. The median length of stay in the program for this treatment group was 4.4 months. The comparison group included the remaining 163 youth (22%) who were referred to MST but did not have the opportunity to complete treatment as a result of reasons unrelated to case progress. Of these, 94% (n = 154) were removed from treatment due to administrative or funding/referral reasons and 6% (n = 9) moved out of the program's service area. The median time for the comparison group from initial program contact to program discharge was 1.45 months.

The study used propensity score matching to balance pre-treatment characteristics of youth across treatment conditions. After comparing multiple methods of matching, the authors settled on the Kernel-based matching approach with a bandwidth of 0.08 and a common support restriction. The covariates included age, gender, race, number of prior episodes of out-of-home care, number of placements, history of maltreatment, history of adjudication, history of juvenile training school placement, and case assignment prior to or during MST treatment. The matched sample lost 20 participants from outside the common support region.

Assessments/Attrition: The study outcomes were examined over a six-year period after the initial program contact. The retrospective use of pre-existing data meant there was no attrition.

Sample:

The majority (57%) of youth were male, and the average age was 14.9 years at the time of referral. For self-reported race and ethnicity, 49% were White, 14% Black, 29% Hispanic, and 8% other or multiracial. In terms of case history, 36% of youth had a history of maltreatment, and more than half (57%) had a history of out-of-home placement. More than one-quarter of youth had previously been adjudicated (26%) and placed in a juvenile training school (27%).

Measures:

The data came from administrative records of the MST providers and the Rhode Island Department of Children, Youth, and Families. The three outcomes measured the time-to-event for youth of out-of-home placement (i.e., removal from parental custody due to reasons such as child behavior or disability, parental inability to cope, abuse or neglect), adjudication (i.e., court determination that the youth committed a crime or delinquent offense), and placement in a juvenile training school.

Analysis:

The analysis used Cox proportional hazards regression models to examine the risk of removal and delinquency outcomes following initial contact with the program. Youth who did not experience the outcome were censored at the end of the study observation period. For youth who experienced a subsequent MST referral or treatment prior to the end of the study observation period (n = 48), the outcomes were censored at the beginning of the second MST referral for treatment. In addition, youth who turned 18 prior to event occurrence or the end of the study period were censored on their 18th birthday. The covariates included prior history for the outcomes.

Intent-to-Treat: The study used all referred participants who began the program, though 4% were excluded because they experienced the outcome before the program began.

Outcomes

Implementation Fidelity:

No quantitative information.

Baseline Equivalence:

Prior to matching, significant differences were found between the treatment and comparison groups on race/ethnicity, the number of removals from parental custody, and the number of out-of-home placement settings in the first removal episode. Relative to the comparison group, the treatment group had fewer racial and ethnic minority youth, experienced fewer out-of-home care episodes, and had fewer placement transitions in their first episode of care. After matching, no significant differences were found between the treatment and comparison groups on these covariates.

Differential Attrition:

No attrition.

Posttest and Long-Term:

The Cox survival models showed significantly lower risks for all three outcomes - out-of-home placement, adjudication, and juvenile training school - in the intervention group than the comparison group. The hazard ratios ranged from .57 to .61.

Study 23

Summary

The intervention examined in this study adapted MST principles and techniques for youth with an autism spectrum disorder. The study used a randomized controlled trial that assigned 15 youths with an autism spectrum disorder and severe disruptive behavior to an MST condition or a usual-services condition. Measures of conduct problems, internalizing, and aggression were obtained at six months and 12 months after recruitment.

The study found that the MST intervention group showed significantly better family adaptability at six- and 12-months post-recruitment than the usual-services control group.

Wagner et al. (2019) found that, relative to the usual-services control group, the MST intervention group showed significantly better:

• Family adaptability at six- and 12-months post-recruitment.

Evaluation Methodology

Design:

Recruitment: The sample came from families and youths registered at an interdisciplinary academic medical center specializing in the diagnosis and treatment of youths with an autism spectrum disorder. Eligible youths had a confirmed autism spectrum disorder, had recently evidenced severe disruptive behavior, and were 10 to 17 years of age. Of 42 families screened for eligibility, 17 were actually eligible and consented to participate and two withdrew during the assessment phase (due to identified factors that affected their eligibility).

Assignment: The 15 participating families were randomized (with a 1:1 ratio using a computerized random number generator) to the MST (n = 8) or usual community services (n = 7) conditions. The control group received a variety of services, but all focused on the individual youth rather than on the systems in which the youth was embedded.

Assessments/Attrition: Assessments occurred at baseline, six months, and 12 months after recruitment. The completion rates were 87% (n = 13) at six months and 73% (n = 11) at 12 months. The 12-month assessment referred to the period after recruitment rather than after program completion and therefore did not define a long-term follow-up.

Sample:

The sample youths had a mean age of 13.8 years. They were 87.5% boys, 75% white, 6.25% African American, 12.5% Asian American, and 6.25% Hispanic/Latino. About 62.5% lived with two caregivers (e.g., biological parents, stepparents).

Measures:

The outcomes, all from caregiver reports, included measures of youth conduct problems, internalizing, family relations, and peer relations. Reliabilities for the measures were high but came from reports of other studies rather than from the sample.

Analysis:

The analysis used analyses of covariance models with a time-by-group term to test for condition differences in changes from baseline to posttest and follow-up. Baseline outcomes were included as part of the time variable and sometimes as covariates as well. Supplemental analyses replaced missing data by carrying the last observation forward.

Intent-to-Treat: The four families that did not complete all three assessments were excluded from primary analyses, but supplemental analyses estimated program effects for the full sample by imputing missing data.

Outcomes

Implementation Fidelity:

Not examined.

Baseline Equivalence:

Tests showed no differences in demographic characteristics. However, caregivers in the MST condition reported higher levels of caregiver stress, lower levels of family adaptability, and more youth conduct problems than did caregivers in the control condition.

Differential Attrition:

Retention was greater in the MST group than the control group at both follow-ups: 100% versus 71.4% at six months and 87.5% versus 57.1% at 12 months. However, the authors reported that "no between-condition differences on demographic or psychosocial measures were found."

Posttest:

One significant effect emerged in 13 tests: The MST group scored better than the control group on family adaptability. Several marginally significant effects also favored the MST group: youth emotional bonding with peers, intensity of youth aggression, and family cohesion. As reported by the authors, the results remained unchanged when imputing missing data and using the full sample.

Long-Term:

Not examined.