KEEP SAFE

Study 1

Summary

Smith et al. (2011), Kim and Leve (2011), and Kim et al. (2013) conducted a randomized controlled trial that examined 100 girls in foster care from two counties (one urban, one rural) in Oregon. The study randomized participants to an intervention group or a control group that received standard services from local child welfare organizations. Assessments occurred one to two years after posttest and measured internalizing, externalizing, prosocial behavior, health-risking sexual behavior, foster-care placement stability, and substance use.

Smith et al. (2011), Kim and Leve (2011), and Kim et al. (2013) found that intervention girls, relative to control girls, demonstrated significant effects for:

• Internalizing and externalizing problem behaviors at six months
• Prosocial behavior combined across the 6-month and 12-month assessments
• Placement stability at the 12-month posttest
• Substance use at 36 months beyond baseline (2 years post-intervention)
• Health-risking sexual behavior at 36 months beyond baseline (2 years post-intervention)
• Small to moderate mediating effects of prosocial, internalizing, and externalizing behaviors on long-term outcomes of substance use
• Small to moderate mediating effect of tobacco and marijuana use on long-term health-risking sexual behavior.

Evaluation Methodology

Design: This randomized controlled trial of the KEEP SAFE (Middle School) program was conducted with girls in foster care from two counties (one urban, one rural) in Oregon. Girls were eligible if they were finishing elementary school between 2004 and 2007 and were referred to foster care through the local child welfare system. Researchers recruited girls and their foster parents from the pool of eligible participants (N=145). Eligibility criteria required that girls not yet be reunited with their original family and that the foster parents provide consent. Recruitment occurred on a rolling basis and stopped when enrollment reached 100. The participants were randomly assigned to the intervention (n=48) and the control condition (n=52) which consisted of the usual services provided by the local child welfare organizations.

After randomization the girls and their foster parents were assessed at baseline (T1) and at 6 months post baseline (T2). The girls were assessed again at 12-, 24-, and 36-months post baseline, which represent the periods shortly after the full program ended (T3), one year after the program ended (T4), and two years after program ended (T5), respectively. The baseline assessment consisted of a standardized interview and questionnaires for each girl and foster parent, an interview with the girl's caseworker, and the collection of child welfare records. The interviews lasted approximately two hours and were conducted in person by assessors who were blind to group assignments. The data collection instruments were designed to measure child and family characteristics, child behaviors, and parenting practices.

The summer component of the program was implemented in the summer before commencement of middle school. The mean time between baseline (T1) and the 6-month follow-up (T2) was 147 days (s.d.= 45.6). When changes in placement/caregiver occurred between baseline and T2 (20% of sample), the girl was followed to her new placement and the girl's new caregiver was recruited to participate in the study. The authors did not report when the transitions to new caregivers occurred.

Booster sessions took place during the first year of middle school. On average, parents attended ten weekly meetings between the completion of the summer session and the T2 assessment. The authors did not report on the specific frequency of the girls' attendance in follow-up sessions, but "participation rates mirrored those of their caregivers" (Kim and Leve, 2011).

The baseline (T1) sample size for the intervention group was 48 and subsequent sample sizes were 48, 47, 44, and 45, respectively, for T2, T3, T4, and T5. Of the three subjects that were lost, one could not be located and two refused to participate. The baseline (T1) sample size for the control group was 52 and subsequent sample sizes were 50, 50, 48, 45, respectively, for T2, T3, T4, and T5. Of the seven subjects that were lost, three could not be located and four refused to participate. Thus, attrition was about 10% percent from baseline to the 2-year post-intervention follow-up.

Sample Characteristics: The average age of the girls at baseline was 11.5 years. The ethnic distribution was 63% European American, 10% Latino, 9% African American, 4% Native American, and 14% multiracial. At baseline, the girls had been in foster care an average of 2.9 years and had experienced an average of 1.4 out-of-home placements. In this sample, 56% of the girls had at least one documented incident of physical abuse, 67% had at least one documented incident of sexual abuse, and 82% had at least one documented incident of physical or sexual abuse.

Measures: Measures were divided into categories of short-term outcomes, long-term outcomes, and predictors.

For short-term outcomes, the Parent Daily Report Checklist (PDR) was used to measure internalizing problems, externalizing problems, and prosocial behavior. This 34-item measure of child behavior problems was administered by telephone individually to foster parents and girls on three consecutive or closely spaced days. In each call, subjects were asked about the preceding 24-hours to improve reliability of the measure. The PDR has been shown to be both valid and reliable.

• Internalizing problems: An internalizing problems composite was computed based on five of the PDR items. The scores were averaged across the three telephone calls at T2 (alpha=.72 for parents and .74 for girls). The parent and girl scores were significantly correlated and were thus combined into a composite internalizing problems score at 6 months. At two years, the report was by caregivers only.
• Externalizing problems: An externalizing problems composite was computed based on 18 of the PDR items. The scores were averaged across the three telephone calls at T2 (alpha=.85 for parents and .81 for girls). The parent and girl scores were significantly correlated and were thus combined into a composite externalizing problems score at six months. At two years the report was by caregivers only.
• Prosocial behavior: A prosocial behavior composite was computed based on 11 of the PDR items. The scores were averaged across the three telephone calls at T2 (alpha=.74 for parents and .75 for girls). The parent and girl scores were significantly correlated and were thus combined into a composite prosocial behavior score.

Placement changes in foster care were measured from welfare system records as the sum from the start of the study through posttest. The number of placement changes ranged from 0 to 7 during this period, and the mean was .56.

Long-term outcomes included substance use, delinquency, and health-risking sexual behavior.

• Substance use: Three indicators were used to assess substance use at T5: How many times in the past year have you (a) smoked cigarettes or chewed tobacco; (b) drunk alcohol; and (c) used marijuana. The response scale ranged from 1 (never) through 9 (daily). Tobacco and marijuana use were combined to create a two-item composite score. The authors did not report on reliability or validity for this measure.
• Delinquency: Delinquency was assessed with two indicators. First, girls' own delinquent behavior was measured with the 36 items from the general delinquency scale, which is a part of the Self-Report Delinquency Scale (SRD). Girls were asked how many times in the past year they had committed various acts (e.g., damaging or destroying property, stealing). The mean of the frequencies across the items was used to represent the level of delinquency. Internal reliability of this measure with this sample was alpha=.85. Second, the girls' association with delinquent peers was measured with 30 items from a modified version of the general delinquency scale from the SRD. Girls were asked how many of their friends were involved in delinquent acts (26 items scored 0="none" to 4="all") and how often their friends used alcohol, tobacco, and marijuana (4 items scored 0="none" to 4="a lot"). Because of different response scales for the first 26 items and the last 4 items, the two item sets were each standardized before combining to compute a mean value. The internal reliability of this scale was alpha=.96.
• Health-risking sexual behavior: Eight items from the girls' in-person interviews were used to assess health-risking sexual behavior at T5. The girls reported on items such as touching a boy's body above or below the waist, having sexual intercourse, having sex with someone who they just met, or having sex with someone using drugs in the past 12 months. Positive answers to these items were totaled to represent the cumulative number of health-risking sexual behaviors. Internal reliability of the measure was .67.

Predictor measures included:

• Maltreatment History: The girls' cumulative maltreatment history at T1 was coded from child welfare case files using a modified version of the Maltreatment Classification System. Trained coders examined these case files to identify incidents of physical and sexual abuse. Two-thirds of the files were double-coded to compute interrater agreement. Agreement on the identification of the number of physical and sexual abuse incidents was high (85.7% and 86.2%, respectively). The maltreatment history variable consisted of the sum of the number of incidents prior to the study.
• Pubertal Development: Pubertal development was measured at T1 using girl and foster parent reports on the Pederson Scales of Development. This measure has been shown to be reliable and valid. The Pederson Scales of Development uses a 4-point Likert scale to rate items such as body hair changes, skin changes, breast growth, and changes in height. Menstruation was measured as a dichotomous variable. A pubertal development score was computed using parent data and girl data. These scores were highly related and therefore averaged to create a mean pubertal development score.

Analysis: For the 6-month assessment, Smith et al. (2011) ran separate stepwise hierarchical linear regression models, one for each T2 outcome (internalizing, externalizing, prosocial behaviors). The first step consisted of the general predictor variables (baseline age, maltreatment history, pubertal development, and behavior) and the second step included the intervention dummy. The study reported beta coefficients, but did not state that the coefficients were standardized (authors state in personal correspondence that coefficients were standardized).

The two-year posttest follow-ups (Kim & Leve, 2011; Kim et al., 2013) conducted structural equation modeling. To use the full intent-to-treat sample (n=100), full information maximum likelihood (FIML) estimation was used. This method has been shown to provide unbiased estimates when data are missing at random. Little's MCAR test indicated that the data were, in fact, missing completely at random. The composite tobacco and marijuana use variable was log transformed and the number of placement changes was square root transformed to more closely resemble normal distributions. Baseline outcomes of substance use, delinquency, and health-risking sexual behavior were not measured at baseline or used as predictors.

Outcomes

Fidelity: The interventionists were supervised weekly, and would regularly be shown videotapes of sessions and provided with feedback regarding adherence to the clinical model. The authors did not report how well the interventionists adhered to these guidelines. Caregivers attended 5.62 of the 6 summer sessions, and 20 of the 40 follow-up sessions during the year offered, on average. Participation rates of girls in the summer sessions mirrored those of their caregivers. Of the 40 follow-up sessions offered to youth, average attendance was 56.4%.

Baseline equivalence: Kim and Leve (2011) found no significant differences on demographic characteristics, age at first placement, number of placements, foster care type, history of delinquency, history of special services, internalizing behavior, or externalizing behavior. Only the baseline measure of severity of neglect differed significantly, but it had no influence in any of the models.

Differential attrition: The authors did not report on the characteristics of attriters vs. non-attriters, yet Little's MCAR test indicated that data were missing completely at random.

Posttest: Smith et al. (2011) examined internalizing problems, externalizing problems, and prosocial behavior at about six months after baseline (conclusion of the summer component, but still during booster sessions occurring in the first middle school year). The program was significantly related to a decrease in internalizing (beta = -.28, p<.01) and externalizing (beta = -.21, p<.01) problems, but not significantly related to prosocial behavior.

Kim and Leve (2011) examined placement changes, internalizing behavior, externalizing behavior, and prosocial behavior at 6-month and posttest combined. Girls in the intervention had significantly fewer placement changes (Cohen's d = .50) and significantly greater prosocial behavior (Cohen's d = .46). There were no significant differences on externalizing and internalizing symptoms (Cohen's d = .02).

Long-term: In Kim and Leve (2011), the intervention group reported moderately lower levels of substance use (composite of tobacco, alcohol, and marijuana) than girls in the control group (Cohen's d = .47) at the two-year posttest follow-up. When the substances were examined separately, the program did not have a significant effect on alcohol use but did significantly affect tobacco use (Cohen's d = .45) and marijuana use (Cohen's d = .57). Delinquency (a composite of the girl's own delinquency and her association with delinquent peers) was marginally significantly lower for intervention girls.

In Kim et al. (2013), girls from the intervention showed significantly lower levels health-risking sexual behavior (Cohen's d = .48) than the control group. Post-hoc analysis indicated that only 4.4% of the girls in the program, as opposed to 17.8% of the girls in the control condition, reported having sexual intercourse in the past year (chi-square = 4.05, df = 1, p = .04).

Mediation: First, the program had significant indirect effects (beta=-.04) on substance use through increased positive prosocial behavior at T2 and T3 and reduced internalizing and externalizing behaviors at T3 and T4. Second, the program did not have a direct effect on delinquency but did have an indirect effect (b=-.07) on delinquency through increased prosocial behavior at T2 and T3. Third, the program had a significant indirect effect (beta=-.12) on health-risking sexual behavior through reduced tobacco and marijuana use.