Positive Action

Study 1

Summary

Flay et al. (2006), Beets et al. (2009), Snyder et al. (2010, 2012, 2013) and Washburn et al. (2011) used a cluster randomized controlled trial that assigned 10 matched pairs (N=20) of schools to intervention and control conditions. First and second grades students were followed through fifth and sixth grades; students who moved to another school were dropped and students who entered a school during the study were added to the sample. Administrative measures were collected on daily absences, suspensions, grade retention, and academic achievement. Students reported on their substance use and violent behaviors, and attitudes toward positive behaviors.

The study found that compared to the control condition, students and schools in the intervention condition showed significant:

•  School-wide reductions in grade retention, suspensions and absenteeism and school-wide improvements in reading and math after three program years, and these results were maintained through one-year post-implementation.
• Lower likelihood at fifth grade to have engaged in self-reported substance use, violence, and sexual activity.

Significant Program Effects on Risk and Protective Factors:

• Better youth social interaction skills
• Improved school supportiveness
• Improved school quality at one-year post-intervention.

Evaluation Methodology

Design: This evaluation was conducted in Hawaii, utilizing a clustered randomized design in one large school district. First and second grade students were followed from spring of 2002 through spring 2006 (grades 5 and 6) by the wave 5 follow-up. Elementary schools within that district, on the islands of Oahu, Maui, and Molokai, were eligible to participate in the design if they were public K-5 or K-6 schools and not academy, charter, or special education schools. Schools were also required to have at least 25% of population eligible to receive free or reduced price lunches, to be in the lower 3 quartiles in standardized test scores, and to have annual student stability rates over 80%. One hundred eleven of 151 elementary schools on these islands were eligible (73.5%).

Schools were stratified by risk score, calculated considering 1) demographics such as school size, mobility of students, ethnicity, and student/teacher ratios, 2) student characteristics such as special education and gifted status, and 3) student behavior and performance such as disciplinary referrals, suspension rates, and standardized achievement scores. There were 19 resulting and usable strata, each containing 3 to 6 schools. Prior to recruitment, schools within these strata were randomly assigned to either the treatment or control condition. If a school declined to participate in its assigned condition, researchers attempted to replace the declining school with one of the remaining schools in the same strata. However, every time a school declined, the remaining schools in the stratum also declined (Flay et al., 2006). Nine strata were abandoned because of lack of consent. This resulted in 10 matched pairs of schools, 5 on Oahu, 3 on Maui, and 2 on Molokai. Compared to schools in abandoned strata and to those who declined, participating schools were not significantly different on variables used to determine risk scores. Furthermore, the treatment and control schools were statistically similar to one another overall and did not significantly differ from the averages of the discarded sets. As planned, trial schools were of higher risk and more stable than the average of all Hawaiian public schools but also represented the schools willing to adopt the program.

Intervention schools were offered the complete PA program free of charge and control schools were offered a monetary incentive during the randomized trial and the PA program upon completion of the trial. Three of the 10 control schools chose to receive the PA program after the formal trial; they were treated as controls at the follow-up to the present study, as anecdotal evidence suggests that they did not fully implement the program, and it is likely that schools need several years to fully implement a comprehensive program to see substantial benefits.

The study followed students who were in first or second grade at baseline (the 2001-2002 academic year) and who stayed in the study schools through fifth or sixth grade (the 2005-2006 academic year). Students who left the study schools during the study period were dropped, and students who joined study schools during the study period were added to the study. Thus, the study included students who entered the schools at any year during the course of the study and who were in fifth grade at the end of the study.

Student self-reports of their behavior were collected at five time points, on each of two cohorts (first graders and second graders at the start of the project). Data were collected for baseline at the end of the academic school year in half of both the control and PA schools and at the beginning of the next school year in the others. The remaining four waves of data were collected at the next four springs. Researchers rather than school staff collected the data.

Because some children changed schools, were sometimes absent for an administration of the questionnaire, or refused to answer selected items, there were missing data at all waves. For student reports of their own behavior, 1,544 students responded at the first wave, 2,116 at the second wave, 1,498 at the third wave, 1,493 at the fourth wave and 696 at the final wave. The sharp drop at the final wave was because 6 of the 20 schools (3 control and 3 PA) did not contain sixth grade and the entire older cohort in those schools was lost to follow-up. There were a total of 7,347 observations from 2,646 children distributed over 20 schools, with an average of 2.8 waves of data for each student.

Beets et al. (2009) reported results from fifth grade students (aged 10-11 years). They were asked to obtain active parental consent and to provide verbal assent to respond to items asking about substance use, violent behavior, and sexual activity. This request garnered responses from 976 intervention students (50% girls) and 738 control students (50% girls), a response rate of 86%.

Sample Characteristics: The sample was evenly split between boys (50.2%) and girls (49.8%). Program schools were 32.7% Hawaiian, 20.5% White, 1.4% Black, 0.2% Native American, 2.7% Pacific Islander, 4.3% Japanese, 3.7% other Asian, 12.4% other ethnicity, and 22.0% multi-ethnic. Control schools were 26.5% Hawaiian, 11.4% White, 2.0% Black, 0.3% Native American, 3.2% Pacific Islander, 11.8% Japanese, 5.1% other Asian, 14.4% other ethnicity, and 25.3% multi-ethnic. Eighty-eight percent of the population of both program and control schools were said to be stable in terms of mobility. Per capita income was about $14,000 for both program and control schools. Sixty-two percent of the population of the program schools were eligible for free or reduced price lunches, along with 56% of the population of control schools. About 11% of both groups were special education students and about 9% were non-English speakers.

Measures: School-level data on daily absences, suspensions, retention in grade, and academic achievement indicators including grade 5 reading and math and grade 4 reading and math were available, as well as student self-report for attitudes toward positive behaviors and lifetime substance use and violence behaviors. Teachers also reported in years 4 and 5 on child behavior, including violence. In order to measure school quality change (Snyder et al., 2012), archival school-level data were obtained from the Hawaii Department of Education Accountability Resource Center Hawaii as part of the state's school quality survey (SQS) accountability system. These data were collected from teachers, parents, and students every 2 years. There were 9 indicators of school quality: 1) safety and well-being, 2) involvement of parents, 3) satisfaction of parents, students, and teachers, 4) quality of student support, 5) focused and sustained action, 6) responsiveness of the system, 7) standards-based learning, 8) professionalism and capacity of the system, and 9) coordinated team work.

Analysis: Analyses vary according to the specific report examined, but include growth curve analysis conducted on the first 4 waves of data, two-level random effects models with students nested within schools, and Poisson models. For the school quality analysis, school quality composite scores (SQC) were created for teachers, parents, and students by calculating the average of all SQS indicators for each respondent group. Primary analysis included matched-paired t-tests, and sensitivity analysis was conducted using permutation tests with Stata v11 permute. Effect sizes were calculated using Hedges' adjusted g. A percent relative improvement (RI) was calculated to better interpret the effect size indicators.

Outcomes

Implementation Fidelity: On average, 85% of teachers completed implementation reports at the end of each unit. In the first year, 59% of the teachers reported completion of the expected 4 or more lessons of Positive Action each week and this increased to 71% by the third year. Implementation fidelity, overall, varied widely between schools, especially in the first year. By the third year, five schools were implementing at a high level of fidelity (though still not at full fidelity, meaning that very few schools achieved much in regards to the family- or community-involvement components). Three were implementing at a moderate-to-high level and the last two schools were still implementing at low levels of fidelity.

Baseline Equivalence: The studies reported no significant baseline differences between conditions for the following list of variables.

• Beets et al. (2009): the percentage of students receiving free or reduced-price lunch, school size, percentage of student stability, and student ethnic distribution; additional characteristics of the school (student-teacher ratio and expenditures per student); characteristics of student populations (proportions of gifted, special education, and English as a second language students); and indicators of student behavioral and school performance outcomes (disciplinary referrals, suspension rates, and standardized achievement scores).
• Beets et al. (2009): self-reported and teacher-reported negative student behaviors such as gets into fights, threatens others, physically hurts others, and hits others (using random-effects models with students nested within schools).
• Washburn et al. (2011): endorsement of positive behaviors (using a simple t-test and a simple linear growth model).
• Snyder et al. (2010): 19 sociodemographic variables and 7 academic outcome variables.
• Snyder et al. (2012): a school quality composite score from teachers, parents, and schools.

In addition, a February 2012 response from Dr. Flay to questions from the Blueprints Board reported on baseline differences on 43 attitudinal variables, with only one significant effect emerging.

Differential Attrition: The analysis of school-level outcomes in Snyder et al. (2010, 2012) had no attrition. The other two papers did not present a full analysis of differential attrition but offered relevant evidence.

Beets et al. (2009) compared the negative behavior scale for control group students who began at baseline and participated in all five years of the study with those who entered the study after baseline. The results indicated no significant difference.

Washburn et al. (2011) noted that there were missing data at each wave and that the loss of data at the last wave came largely from six schools without a sixth grade and absences of students on the day of the assessments. They also stated, "Given that parents, not students, usually decide if a student is in a school or not and, therefore, missingness is not related to the student behavior outcome (in only two cases was missingness significantly correlated with the outcome and in both cases the correlation was small, only -0.22 and -0.10) ..."

Individual-Level Fifth Grade Outcomes (Beets et al., 2009): Individual-level analysis at the end of fifth grade using two-level models for students, but without baseline controls, showed that, relative to the control group, the intervention group had significantly lower substance use, violent behavior, and sexual activity according to self-reports, and lower violence according to teacher reports. Relative risk and odds ratios ranged from .54 for teacher-reported violence to .24 for self-reported sexual activity.

School-level analysis of mean differences at the end of fifth grade showed significantly lower means for intervention schools on self-reported substance use and violent behavior and teacher-reported violent behavior (self-reported sexual activity and teacher-reported substance use were marginally significant).

A dose-response analysis of students showed that program effects were larger for those students exposed to the program 3-4 years compared with those exposed for only 1-2 years.

Individual-Level Growth Models in Endorsement of Positive Behaviors (Washburn et al., 2011): Three-level growth models of a scale of positive behaviors revealed a significant interaction between year and condition and between year squared and condition. The interactions showed that the number of positive behaviors endorsed decreased from year to year and this decrease was significantly lower in the intervention group. By the fifth wave, the sample means were 50.88 and 37.23 for the intervention and control schools, respectively. Cohen's d for the final wave (controlling for baseline differences) was 0.46.

School Academic Outcomes (Snyder et al., 2010): At posttest, the mean comparisons indicated that intervention schools had significantly higher math and reading scores, significantly lower absenteeism, and marginally fewer suspensions. After completion of the randomized trial, at one-year post trial as intervention schools continued to implement the program, reading TerraNova and math and reading scores were significantly higher among intervention schools; and absenteeism and suspensions were significantly lower for intervention schools. All of the effect sizes were moderate to large, regardless of the level of significance.

Random intercept growth models largely confirmed the results for math scores, reading scores, and absenteeism, while they showed some significant intervention effects on suspensions and retention that were absent in the mean comparisons.

School Quality Results (Snyder et al., 2012): By 1-year post-trial, school-quality composite (SQC) scores among all respondent groups participating in Positive Action increased significantly compared to scores among the control group schools (which actually exhibited decreases). In fact, SQC scores demonstrated by the PA schools exceeded even the statewide averages. Effect sizes were greatest among teachers (1.61) and also large among parent (1.26) and student (1.31) reports. Relative improvements on SQC scores among the intervention schools ranged from 16.2% to 21.1% across the three respondent groups. When analyzed individually, the majority of the 9 indicators of school quality were found to have improved across all respondent groups when compared to the reports of the control school participants. Almost all effect sizes were moderate to large. Time x condition interactions for teacher, parent, and student models were all statistically significant, also bearing support for the outcomes indicating significant improvement among intervention schools in school quality due to the Positive Action implementation.

Mediation Analysis (Snyder et al., 2013): Using structural equation models, the analyses first demonstrated direct intervention benefits for student-reported measures of positive academic behavior, substance use, violent behavior, and sexual activity. In tests for mediation, the models found significant indirect effects of the program via positive academic behavior on substance use, violent behavior, and sexual activity. Teacher reports on the same measures of student behavior, except for sexual activity, also demonstrated significant direct and indirect effects.

Study 2

Summary

Bavarian et al. (2013, 2016), Duncan et al. (2017), Lewis et al. (2012 [certified], 2016), Lewis, DuBois et al. (2013), Lewis, Schure et al. (2013), Li et al. (2011 [certified]), Silverthorn et al. (2017), SCDRC (2010), and Washburn et al. (2011) used a cluster randomized controlled trial that assigned 7 matched pairs (N=14) of schools to intervention and control conditions. Third grade students followed through eighth grade; students who moved to another school were dropped and students who entered a school during the study were added to the sample. Measures of substance use, violence, bullying, conduct problems and academic performance were collected from students, parents and teachers.

The study revealed that, compared to the control condition, the students and schools in the intervention condition showed significantly:

• Higher socio-emotional and character development at grades 5 and 8
• Lower self-reported substance use at grades 5 and 8
• Lower self-reported violence at grades 5 and 8, lower parent-reported bullying at grade 8, and lower self-reported bullying at grades 5 and 8
• Lower depression and anxiety at grade 8
• Lower unhealthy food consumption at grade 8
• Lower school-level disciplinary referrals and suspensions at grade 8
• Better reading test scores at grade 8.

Significant Program Effects on Risk and Protective Factors:

• Lower disaffection with learning and higher teacher-rated academic motivation at grade 8
• Lower normative support for aggression at grade 8
• Higher life satisfaction at grade 8
• Better social interaction skills
• Higher school and peer self-esteem at grade 8.

Evaluation Methodology

Design: This evaluation was conducted in the Chicago Public Schools system beginning in the fall of 2004. Four hundred eighty-three K-6 and K-8 schools were screened for eligibility on a number of criteria. Schools were excluded if they were not community schools (if they were academy, charter, or special education schools), if they were already using Positive Action or a similar program, if their enrollment rate was below 50 or above 140 students per grade, if their annual student mobility rates were 40% or above, if more than 50% of students passed the Illinois State Achievement Test, and if less than 50% of their students received free lunches. According to these criteria, 68 schools were invited to attend information sessions about the PA program and research study. Representatives from 36 of these schools attended one of the information sessions. Eighteen of the schools agreed to participate with the understanding that they would be matched with another suitable school in the pool and randomly assigned to conditions. Using a SAS program, they were matched into pairs on a range of variables, including ethnicity (percentage White, Black, Hispanic, and Asian), ISAT test scores, attendance rate, truancy rate, socioeconomic variables, percentage of students who enrolled or left school during the school year, number of students per grade, percentage of parents reported to demonstrate school involvement, and percentage of teachers employed by the school who met minimal teaching standards. Using this process, the best seven of nine well-matched pairs of schools were selected, and thus four schools were dropped, leaving 14 schools. These 14 schools were not significantly different from the 68 eligible schools on any of these measures, nor were treatment and control schools different. The study followed a single cohort of students who were in the third grade when the program implementation began. New students joining this cohort in subsequent years were also included and followed.

Each member of the seven pairs was randomly assigned to either the treatment or a waitlist-control condition (who would receive the program after three years). The treatment condition schools began program implementation in the 2004-2005 school year (and the waitlisted control schools were to begin implementation in the 2007-2008 school year, but were then asked to hold off for another 3 years so that the study could continue through grade 8). During Phase I of the funding, five assessments were conducted at baseline (Fall 2004), Spring 2005, Fall 2005, Spring 2006 and Spring 2007 (at the end of grade five). Every student who was present in the study schools at each wave of data collection was assessed but students who moved out of the study schools were not tracked. As such, the sample is slightly different at each assessment point. At the end of the three-year study period, approximately 510 fifth graders completed the questionnaires; slightly more than half of these students (290, 56.9%) were part of the original sample of 590 students at baseline. Li et al. (2011) found no significant difference on baseline problem behaviors between stayers and dropouts across the multiple imputations. Also, the researchers stated via email communication that there was no differential dropout or addition by group condition.

For the five-year follow-up, Duncan et al. (2017) and Lewis et al. (2016) reported on eight assessments from grade 3 to grade 8. The sample again lost students who moved but added students new to the schools. A total of 1,130-1,170 students in the 14 schools had at least one data observation. However, Duncan et al. (2017) noted that, because of condition uncertainty, students "who were only ever in the study at the fall of either the 4th or 7th grade" were excluded.

Sample Characteristics: Students in program schools were, on average, 10.23% White, 52.48% Black, 32.24% Hispanic, and 7.18% Asian American. Thirty-four percent met minimal state achievement test criteria and 89% qualified for free lunch. Students in control schools were 11.73% White, 55.35% Black, 28.62% Hispanic, and 4.14% Asian American. Thirty-four percent met minimal criteria on state achievement tests and 91% were eligible for free lunches.

The sample of fifth graders with data at baseline (n=290) were primarily African American (46.4%); the remaining groups were 27% Hispanic, 6.9% White non-Hispanic, 2.8% Asian, and 17% other or mixed. There were 49% girls in the control group and 51% in the intervention group.

Measures: Across the eight waves, outcome measures covered behaviors, values, attitudes, academics, and school discipline. Unless otherwise noted, the measures come from youth self-reports.

Lifetime prevalence of substance use (Li et al., 2011; Lewis et al., 2012). Items asked if subjects had ever (1) smoked a cigarette, (2) drank alcohol, (3) gotten drunk on alcohol, (4) used marijuana, or (5) used other more serious drugs. A count variable measured the number of substances ever used. Alphas ranged from .71 to .79 across the waves.

Lifetime prevalence of serious violence-related behavior (Li et al., 2011; Lewis et al., 2013). Starting at the end of 5th grade, items asked if subjects had ever (1) carried a knife, (2) threatened to cut or stab someone, (3) cut or stabbed someone on purpose, (4) been asked to join a gang, (5) hung out with gang members, or (6) been a member of a gang. A count variable measured the number of behaviors ever having engaged in. Alphas ranged from .74 to .82 across the waves.

Bullying (Li et al., 2011; Lewis et al., 2013). Bullying behaviors were measured using six items selected from the Aggression Scale. Children responded to how often in the past 2 weeks they had engaged in verbal or physical aggression at school (e.g., teased others, shoved others). Responses ranged from 0 (never) to 3 (many times) and, after dichotomizing into ever and never, were summed to create a count variable. Alphas ranged from .83 to .90 across the waves.

Disruptive behaviors (Li et al., 2011; Lewis et al., 2013). Children were asked to respond to six items about how often in the past couple of weeks they had engaged in different problem behaviors at school (e.g., taking something at school that belonged to others, skipping class). Responses ranged from 0 (never) to 3 (many times) and, after dichotomizing into ever and never, were summed to create a count variable. Alphas ranged from .77 to .81 across the waves.

Social-Emotional and Character Development (SECD) (Lewis et al., 2012; Duncan et al., 2017). The 28-item Child SECD Scale was adapted from multiple existing measures of social skills. An average composite score of the 28 items was created for each of the eight waves, where higher scores indicate higher SECD skills. Example items are: "I try to cheer up other kids if they are feeling sad", "I apologize when I have done something wrong", "I speak politely to my teacher", "I keep my temper when I have an argument with other kids", "I listen (without interrupting) to my parents", and "I follow school rules". Responses to these items on a 4-point scale allowed students to indicate how often they performed each SECD-related behavior (1= none of the time; 2= some of the time; 3= most of the time; and 4= all of the time). Alphas ranged from .88 to .92 across the waves.

Normative beliefs supporting aggression (Lewis et al., 2013). Students answered questions adapted from the Normative Beliefs About Aggression Scale, which has established reliability and validity for school-aged children. Eight items (e.g., Is it ok or wrong to hit, shove, yell, fight other people?) were rated on a 4-point scale (really wrong to perfectly ok) and averaged to create a composite score, with higher scores reflecting the belief that aggression is more acceptable (alpha range .81-.93). Given a skewed distribution of responses, the scale score was split for analysis using a median split across all waves.

Parent-reported bullying (Lewis et al., 2013). Parents responded to six items (alpha range .73-.83) regarding bullying (e.g., hits others, teases, threatens to hurt others) in the past 30 days. The items used a 4-point scale (never to almost always) but were dichotomized and converted to a count. The outcome was assessed at Waves 1-5 and Wave 8.

Parent-reported conduct problems (Lewis et al., 2013). Parents responded to seven items (alpha range .74-.81) regarding conduct problems (e.g., truancy, cheating, stealing) in the past 30 days. The items used a 4-point scale (never to almost always) but were dichotomized and converted to a count. These outcomes were assessed at Waves 1-5 and Wave 8.

Disaffection with learning (Bavarian et al., 2013). Four items from a measure of student engagement used a four-point Likert scale ("Disagree A LOT" to "Agree A LOT") and statements such as "When I'm in class, I think about other things" and "When I'm in class, my mind wanders". A composite with high scores reflecting more disaffection had alphas ranging from .64 to .71 across the eight waves.

Academic grades (Bavarian et al., 2013). Students were asked, "What grades have you been getting this school year?" Response options ranged from 1 to 9 (e.g., 1 = Mostly Fs, 4 = mix of Cs and Ds, and 9 = Mostly As).

Teacher-assessed academic ability (Bavarian et al., 2013). Teachers rated students on reading, mathematics, academic performance, and intellectual functioning using a 5-point Likert scale (1 = Far below grade level to 5 = Far above grade level). A composite score indicating higher ratings of students' academic ability had alphas ranging from .97 to .98.

Teacher-assessed academic motivation (Bavarian et al., 2013). A single-item measure used response options ranging from "Extremely low" to "Extremely high".

School-level disciplinary referrals (Lewis et al., 2013). School-level aggregated data reported on the school district's website were accessed for school years 2002/2003 to 2009/2010. Disciplinary referrals were based on a range of disruptive, bullying, and illegal student behaviors, the latter of which included (but were not limited to) vandalism, assault, theft, and possession of drugs or dangerous weapons. Analyses on school-level data were adjusted for school size by including it as an exposure variable in the model.

School-level suspensions (Lewis et al., 2013). The measure came from the same source and used the same procedures as for disciplinary referrals.

School-level standardized reading scores (Bavarian et al., 2013). Archival reading scores of non-English Language Learners came from a standardized, school-administered, statewide test (the ISAT). A single weighted average of the percentages of students falling into four achievement levels was created for each school overall and by demographic subgroups. A value-added metric index of ISAT performance reported by the school district was used to control for the prior year ISAT scores of students. Data were available for the cohort transitioning from grades 7 to 8 (2009-2010).

School-level standardized math scores (Bavarian et al., 2013). The standardized math scores also came from the ISAT.

School-level absenteeism rates (Bavarian et al., 2013). The school district reported average daily attendance rates for each school on a scale from 0 to 100%; these statistics were converted to a measure of average daily absenteeism by subtracting 100 from each school's respective year-end attendance.

Positive behaviors (Washburn et al., 2011). A total of 51 behavior items were asked, each with the same four response options: "none of the time," "some of the time," "most of the time," or "all of the time." The options were coded 1 for or "all of the time" and 0 otherwise, summed, and transformed into a percentage of maximum possible score.

Positive affect (Lewis, DuBois, et al., 2013). Positive affect was measured using a modified 6-item version of the Positive and Negative Affect Scale for Children. Students reported the extent to which they had experienced each type of feeling (e.g., excited, happy) in the last 2 weeks using a 4-point scale ranging from "None of the time" (1) to "All of the time" (4). Alphas ranged from.70 to .87 across time points.

Life satisfaction (Lewis, DuBois, et al., 2013). The measure consisted of 3 items: "My life is just right", "I have a good life", and "I have what I want in life". Students indicated how much they agreed with each statement on a 4-point scale ranging from "NO!" (1) to "YES!" (4). Alphas ranged from .71 to .84 across time points.

Depression (Lewis, DuBois, et al., 2013). Assessed at waves 5 through 8, the measure used six items from the Behavior Assessment System for Children, such as "I feel depressed." Alphas ranged from.70 to .79.

Anxiety (Lewis, DuBois, et al., 2013). Assessed at waves 5 through 8, the measure used six items from the Behavior Assessment System for Children, such as "I often worry about something bad happening to me." Alphas ranged from .70 to .79.

Healthy food consumption and exercise (Bavarian et al., 2016). The multi-item scale available at all waves averaged responses to questions about how much of the time they "eat fresh fruits and vegetables", "drink or eat dairy products", and "exercise hard enough to sweat and breathe hard." The items, which allowed for responses ranging from 1 = none of the time to 4 = all of the time, loaded together in a factor analysis.

Unhealthy food consumption (Bavarian et al.,2016). The multi-item scale available at all waves averaged responses to questions about how much of the time they "eat junk food (chips and candy), "eat fast food", and "drink soda pop." The items, which allowed for responses ranging from 1 = none of the time to 4 = all of the time, loaded together in a factor analysis.

Personal hygiene (Bavarian et al., 2016). The multi-item scale available at all waves averaged responses to the following statements: "I wash my hands after using the toilet", "I brush my teeth at least twice a day", and "I cover my nose and mouth when I sneeze". The items, which allowed for responses ranging from 1 = none of the time to 4 = all of the time, loaded together in a factor analysis.

Consistent bedtime (Bavarian et al., 2016). For this single-item measure, students at all waves rated how much of the time they "go to bed by 9:00 PM on school nights." The measure was re-categorized as dichotomous, (0 = Not all of the time and 1 = All of the time) based on research suggesting school-age children need 10-11 hours of sleep each night.

Body Mass Index (BMI) percentile (Bavarian et al., 2016). Measures of weight and height taken by researchers were used to calculate BMI percentiles, which were then classified as underweight, normal, at risk for overweight, and obese. Due to small sample size (N = 3 students), the underweight category was not included in the analysis, and a dichotomous outcome, unhealthy BMI percentile, was created for analyses, with 0 = healthy weight and 1 = overweight or obese (i.e. BMI percentile ≥ 85%).

Self-esteem (Silverthorn et al. 2017). Self-esteem was measured using the Self-Esteem Questionnaire, which includes self-evaluations of self-worth in different scenarios. An adapted short form more appropriate for use with third grade students, which covered the domains of peer, school, family, appearance, and sports, was administered. Internal consistency was acceptable overall (α=.66-.82) across grades 3-8.

Processes for self-esteem formation and maintenance (Silverthorn et al. 2017). The Self-Esteem Formation and Maintenance Questionnaire is a 21-item measure developed especially for evaluations of the Positive Action program. There are two subscales, adaptive and maladaptive, that assess attitudes and behaviors of youth. The measure was designed to be appropriate for use with third grade students. Internal consistency was acceptable (α=.74-.81).

Misconduct (Duncan et al., 2017). A 12-item scale used child self-reports on aggression, bullying, and delinquent and disruptive behavior. Cronbach's alphas over the eight waves ranged from .86 to .91.

Self (Lewis et al., 2016). Three measures of self-development, self-control, and self-concept came from subscales of the 28-item Social-Emotional and Character Development Scale. These measures have been shown to correlate with outcomes such as substance use, violence, altruism, and life satisfaction.

Peer Affiliations (Lewis et al., 2016). Two measures of prosocial and deviant peer affiliation were created and exhibited expected associations with a range of other outcomes such as substance use, life satisfaction, and depression.

Ethics (Lewis et al., 2016). Measures of positive morality, negative morality, altruism, and empathy came from a reduced and slightly reworded version of the Belief in Moral Order scale and the Children's Empathic Attitudes Questionnaire. The measures have been found to be correlated with theoretically related outcomes.

Social Skills (Lewis et al., 2016). Six measures of prosocial interactions, respect for parents and teachers, honesty, aggressive problem solving, and competent problem solving came from subscales of the Social-Emotional and Character Development Scale and from the Social Skills Problem Solving Measure. These measures have shown evidence of validity in prior research with measures of behavioral adjustment to school.

Analysis: Outcomes related to problem behaviors (Li et al., 2011) were analyzed using multiple imputation to handle the missing values on the covariates and baseline problem behaviors. The strategy of "multiple imputation, then deletion" (MID) used all cases for imputations but then deleted cases with imputed outcome variables. Students who joined the schools after the beginning of the study accounted for the highest proportion of the imputed values on baseline problem behaviors (41.5%) in the analyzed data. To test for baseline differences, multilevel analyses were conducted to examine whether cohort (grade 3) students in the intervention schools were different from students in the control schools on demographics and baseline behaviors. The analyses on baseline problem behaviors were run with and without imputed data. To test for differential attrition, multilevel regressions of baseline problem behavior on stayer versus dropout group membership were conducted using multiple imputed datasets.

To examine program effects, multilevel Poisson models were used with and without imputed data. The three-level models included students (level 1) nested within schools (level 2) nested within pairs (level 3). Introducing a third level random effect partitioned the between-pair variation from the within-pair variation; hence, intervention effects could be tested with greater precisions given that the pairs were well matched. The significance of the program effects in the multilevel models was tested against a standard normal distribution, which assumes a sufficiently large number of schools. Sensitivity analyses using an adjusted degrees of freedom (d.f.=12) were conducted to provide more conservative tests of the program effects for each outcome behavior.

Outcomes related to substance use (Lewis et al., 2012) were analyzed for program effects using three-level multilevel growth curve models (with time nested within individuals, and individuals nested within schools) that controlled for baseline values, and for mediation of program effects using structural equation models. Missing values were handled with Full Information Maximum Estimation, and standard errors were estimated using bootstrap procedures. The models did not control for baseline outcomes, as asking about substance use in grade 3 would have little value. Tests of significance were done at the school-level and based on the sample size of 14, while effect sizes were calculated from the student-level analysis.

Outcomes related to academics (Bavarian et al., 2013) were analyzed using three-level multilevel growth curve models (with time nested within individuals, and individuals nested within schools) for student-level data and two-level multilevel growth curve models for school-level data (time nested within schools). All analyses controlled for baseline values. The models treated condition as a school-level variable. Full information maximum likelihood estimation used all valid observations to model school differences and tested for group-by-time and group-by-time squared interactions. Because of the small Ns (7 per condition), one-tailed p-values were used in tests of effects of the program on school-level outcomes. Gender and student mobility were examined as moderators.

Outcomes related to problem behaviors (Lewis et al., 2013) were analyzed using three-level multilevel growth curve models for student- and parent-reports (time nested within individuals, and individuals nested within schools) and two-level multilevel growth curve models for school-level data (time nested within schools) that controlled for baseline values. The models treated condition as a school-level variable. To handle missing data, full information maximum likelihood estimation was used with logistic regression for binary outcomes and Poisson regression for count outcomes. Gender and student mobility were examined as moderators for student- and parent-reported measures.

Outcomes related to emotional health (Lewis, DuBois, et al., 2013) were analyzed using either growth curve models of change over time or random intercept models of wave 8 outcomes. The models for depression and anxiety (measures available for waves 5 to 8) used the school-level average of student-reported negative affect as a baseline control. Mediated effects were analyzed using structural equation models, with missing values handled by Full Information Likelihood Estimation. Because the structural equations models could not incorporate clustering, supplementary analyses estimated three-level models to account for within-school clustering.

Outcomes related to health behaviors (Bavarian et al., 2016) were analyzed with random-time coefficients in structural equation models. The structural equation models could not be estimated in a multilevel framework, but sensitivity analyses were done that included three levels and tests for the condition by time interaction without structural equation modeling. Multiple imputation was used for the model of BMI percentile.

Outcomes related to self-esteem (Silverthorn et al., 2017) were analyzed with nested growth-curve models, with covariates for condition, time, condition by time, and quadratic terms for time and condition by time. These models inherently adjust for baseline outcomes and schools, the unit of assignment.

For the five-year follow-up assessment, Duncan et al. (2017) examined outcome trajectories using growth mixture models with full information maximum likelihood estimation and robust standard errors. The models adjusted for clustering within schools and controlled for baseline outcomes. The estimation included all eight assessments and allowed for the use of all children with at least one observation. The models estimated latent growth classes that allowed for moderation tests of program effects across the latent classes.

Also for the five-year follow-up assessment, Lewis et al. (2016) estimated two-level and three-level growth curve models that allowed for random intercepts across students but not across schools (the results allowing random intercepts across schools, available on request, were said to be similar). Tobit analysis for skewed outcomes were limited to two-level models of time and student, and linear and logistic regression analysis used three-level models for time, student, and school. Additionally, the Benjamini Hochberg false discovery rate was used to adjust for testing of program effects on multiple measures.

Outcomes

Attrition and Baseline Group Equivalence: PA and control schools were not significantly different on any baseline measures, including child-, parent-, and teacher-reported measures. Differences on ethnicity composition were significant; there were more African-American students and fewer students in the other/mixed ethnicity group for the control condition compared to the PA condition. Multilevel regression analyses, controlling for demographic variables and clustering of students within schools and schools within pairs, showed that PA students reported marginally higher rates of problem behaviors at baseline than control students using the original data (i.e., including stayers only; n~290 with valid responses on baseline problem behaviors). For multiply-imputed datasets (including stayers and newcomers; n~510, the difference was nonsignificant. To control for potential bias, baseline problem behaviors as well as demographic variables were controlled for in the following analyses.

Across the multiple imputations, no significant difference on baseline problem behavior was found between stayers and dropouts controlling for demographic variables and clustering of students within schools and within pairs. Comparisons between stayers and newcomers in the control group on the behavior outcomes also showed nonsignificant differences (substance use, incidence ratio, violence, bullying behaviors and disruptive behaviors). Results suggested that stayers endorsed fewer items on problem behaviors than dropouts and newcomers, although none of the differences were statistically significant.

During the final wave, differences on ethnicity composition were found between the program and the control groups, consistent with the sample characteristics at baseline.

Response rates for teacher surveys ranged from 85 to 100% at all collection points while rates for parent reports were 93% in the first Wave, 77% for Wave 2, 76% for Wave 3, and 72% for Wave 4. (It should be noted that reports of data from baseline through Wave 5 do not include parent- or teacher-reported data.) As for students themselves, researchers provide information on all students who entered study schools at baseline or at any point over the study period, as analysis included all youth who were present in study schools at each wave. In addition to the 593 youth who were present at baseline (47% retained at Wave 5), 88 youth entered at Wave 2 (27% retained at Wave 5), 100 entered at Wave 3 (55% retained at Wave 5), 52 entered at Wave 4 (56% retained at Wave 5), and 108 entered at Wave 5. Thus, 941 students completed at least one local-site survey. Five hundred twelve students participated in the Wave 4 assessment (61% of the 833 who ever participated up to that point) and 500 (53%) participated in the Wave 5 assessment. There is no mention in the report of analyses conducted at Wave 4 or Wave 5 to determine if the program and control groups were still equivalent, but email communication with researchers states that there was no differential dropout or addition by group condition. From waves 6 to 8, there were 392 leavers and 240 joiners (Lewis et al., 2013, supplementary materials). By Wave 8, 363 students (including 131, or 21%, of the original cohort students) remained in the study, reflecting changes in school sizes, consent rates, and the high mobility rate of this population. There were 1170 students with data for at least one wave across all eight waves.

Implementation Fidelity: Schools varied widely in how well they implemented the program. Implementation fidelity measures included teacher-reported amount and quality of classroom PA activities and teachers' perceived effectiveness of and attitudes towards the program, which were measured weekly, by unit, and yearly. Additional measures included year-end school administrator reports and mid-year and year-end student reports of exposure to program activities and attitudes towards the program. By the end of year 3, one school was implementing at a low level (scoring 50% or less on all implementation measures), while 4 were implementing at a moderate level (between 50% and 60%), and two were implementing at a moderate to high level (between 60% and 70%).

Li et al. (2011): After three years of the PA program, results using the MID approach showed students in the PA schools endorsed significantly fewer items for substance use, serious violence, and bullying behaviors. PA students also reported engaging in fewer disruptive behaviors, although this was not statistically significant. The positive program effects can be translated into 31% reduction in substance use behavior, 36% reduction in violence behavior, 41% reduction in bullying behaviors, and 27% (not significant) reduction in disruptive behaviors.

Lewis et al. (2012): The study found that intervention students had significantly lower scores on the substance use composite scale at Wave 8 (sensitivity analyses treating the outcome as a count variable were similar). In addition, the intervention had a significant effect on the slope of the Social and Emotional Character Development Scale (SECD); the slope of scale decreased over time, but the intervention significantly mitigated this decline.

Mediation analysis showed that the intervention had a significant indirect effect on the substance use composite through the SECD scale and that the mediation by SECD eliminated the direct effect of the intervention on the substance use composite (i.e., complete mediation).

Bavarian et al. (2013): The study reported statistically significant program effects for disaffection with learning (one of the two student-report measures; effect size=-0.19; p < .01 two-tailed) and significant effects for academic motivation (one of the two teacher-report measures; effect size=0.39, p < .05 two-tailed). However, the benefit of the intervention for disaffection with learning had disappeared by the end of the study. No significant differences emerged for measures of grades or teacher rated academic performance. Analysis of school-level data revealed marginally significant lower absenteeism in program schools versus control schools (effect size=-0.78, p < .05 one-tailed).

For all students, marginally significant results were found for school-based reading test scores at grade 8 but not for math test scores. African American males improved significantly more on reading (effect size=1.5, p < .05 two-tailed).

Gender appeared to moderate the effects of Positive Action on teacher-rated academic ability with program effects being larger for boys than girls.

Lewis et al. (2013): The condition-by-time interaction coefficients revealed statistically significant program effects for six of the eight problem-behavior measures examined. Three of the four youth-reported measures - normative beliefs supporting aggression (effect size=-0.68, p < .01), bullying (effect size=-0.39, p < .01), and, disruptive behaviors (effect size=-0.50, p < .05) - improved significantly more for program schools than control schools. Gender moderated youth-reported problem behavior with girls demonstrating greater improvement than boys.

Changes in one of the two parent-reported measures, bullying (p<.05), were positively affected by the program. Gender again moderated parent-reported problem behavior but with boys demonstrating greater improvement than girls.

Both school-level outcomes, disciplinary referrals and suspensions, were significantly improved when comparing program schools and control schools (p<.01).

The analysis of the two teacher measures showed no significant program effects.

The results in Table 2 of the article also showed significant interactions of condition by time squared, which may reflect convergence of the conditions over time. To check, Table 3 compared predicted probabilities of the conditions at Wave 8 (though without significance tests). For outcomes with a significant interaction of condition by time squared, the effect sizes at Wave 8 were -.37 for bullying, -.58 for school-level disciplinary referrals, and -.27 for school-level suspensions.

Washburn et al. (2011): The study found a significant year by condition interaction that indicated a positive program effect on students' positive behaviors. Children in intervention schools had a mean score of 63.53 at baseline and children in control schools had a mean of 67.55. By the final wave of the study, children in control schools had a mean score of 39.71, and children in intervention schools had a higher mean of 43.52. Cohen's d at the final wave, controlling for baseline differences, was 0.41.

Lewis, DuBois, et al. (2013). The study of emotional health used all eight waves of data for outcomes of positive affect and life satisfaction and used Waves 5 through 8 for outcomes of depression and anxiety. For positive effect, the intervention by time interaction was marginally significant (p < .10; d = .17). For life satisfaction, the intervention by time interaction and intervention by time-squared interaction were significant, which produced "a notable difference at study endpoint (d = .13) that favored students in PA schools." For depression and anxiety, intervention students reported significantly fewer symptoms at the Wave 8 endpoint (p < .05; d = -.14 and p < .001; d = -.26, respectively).

In addition, mediation analyses showed significant indirect effects of the intervention on each of the four outcomes via the measure of social and emotional character development. The indirect effects occurred for the time slope for positive affect and life satisfaction and for the Wave 8 endpoint for depression and anxiety.

Bavarian et al. (2016). The study of health behaviors showed that the program had a significant negative effect on unhealthy food consumption in the structural equation model but not in the sensitivity checks. It had a marginally significant positive effect on healthy food consumption and exercise. It did not have a significant effect on personal hygiene, but the sensitivity check showed a marginally significant positive effect. The program had no effects on consistent sleep or healthy BMI.

The mediation analyses showed significant indirect effects of the intervention via social and emotional character development on healthy food and exercise and personal hygiene.

Silverthorn et al. (2017). The study of self-esteem found that students in intervention schools showed significantly more positive change over time in measures of peer self-esteem, school self-esteem, and use of adaptive self-esteem formation and maintenance strategies. Intervention students also had a small but significantly greater decline in maladaptive self-esteem processes than the control group, and boys in the intervention group showed a significant decline in the measure of sport self-esteem.

Duncan et al., (2017). At the end point (spring grade 8), children in the intervention schools had significantly higher scores on social and emotional character development and significantly lower scores on misconduct. For the moderation tests, the growth mixture models identified two latent trajectory classes for social and emotional character development and two latent trajectory classes for misconduct. The program had significant effects for all classes except one, and the best models indicated minimal differences in program effects across the classes. The authors concluded that "the program appeared similarly beneficial for trajectories regardless of class membership."

Lewis et al. (2018). Over the five years of assessment (through grade 8), there were significant condition-by-time coefficients for nine of 15 outcomes tested. The intervention group improved significantly more or declined significantly less than the control group on measures relating to self-control, prosocial and deviant peer affiliations, ethics, and social skills.

Mathematical Analysis (SCDRC, 2010)

An independent analysis of the same data by Mathematica found no significant effects. In a letter to the Blueprints Board, Dr. Flay says that the poor results came from 1) evaluation of Positive Action along with six other programs, 2) low statistical power stemming from the small number of schools, 3) examination of only the subset of outcome variables and years available from the full study, and 4) use of models assuming a normal distribution of outcomes. Thus, using different measures, matching analytic techniques to the distribution of the outcomes, and extending outcomes to grade 8 produced positive findings.

Study 3

Summary

Flay et al. (2001) used a quasi-experimental design based on retrospective school-level archival data and matching. The sample included 20 elementary schools in Nevada and Hawaii that had already implemented the program and 40 controls schools that were matched to the intervention schools on demographic characteristics. The outcome measures of student performance and disciplinary referrals/actions came from archival data.

Flay et al. (2001) found that students in the intervention schools compared to students in the matched control schools showed significantly:

• Higher math, reading, and language achievement
• Lower violence

Evaluation Methodology

Design: This study used retrospective school-level archival data to employ a matched-control research design. Two school districts, one in Nevada and one in Hawaii, both with eight or more elementary schools that had implemented PA for three or more years and had easily-available school-level archival data (e.g., student performance and disciplinary referrals/actions), were chosen for the evaluation. Two matched control schools for each program school were selected - matched on the following demographics: percent free/reduced lunch, mobility rates, and ethnic distributions. Data from all (matching and non-matching) non-PA schools were also included in a third category for analysis.

Sample: In the Nevada school district there were 12 PA schools, 24 matched control schools, and 87 non-PA schools (including matched controls). In the Hawaii district there were 8 PA schools, 16 matched control schools, and 117 non-PA schools (including matched controls). The PA and matched control schools in both districts were comparable on all available variables. In Nevada, the PA schools were also similar to all non-PA schools. However, in Hawaii, the PA schools, compared to all non-PA schools, had higher percentages of Japanese/Chinese students, lower percentages of White and Hawaiian students, higher rates of mobility, and lower percentages of students receiving free/reduced lunch.

Measures: School archival data consisted of standardized test scores and disciplinary reports. In Nevada, achievement scores were the average of the 1995-96 and 1996-97 district level Grade 4 percentile ranks on the TerraNova Comprehensive Test of Basic Skills. Disciplinary data consisted of reports of incidents of student-to-student violence, student-to-staff violence, and possession of weapons, for the same two years. Absenteeism rates were also analyzed. In Hawaii, achievement data consisted of the percent of students scoring above average on the Stanford Achievement Test for three school years (1994-95, 1995-96, and 1996-97). Hawaii reports disciplinary data in four categories: felonies, misdemeanors, department rules, and school rules. Number and rates of suspensions and absenteeism rates were also analyzed for Hawaii. Because preliminary analyses found no significant differences across years of data, data across years was combined for all reported analyses.

Analysis: Analysis of covariance was conducted, using the stratifying variables as covariates. For achievement data, multivariate analysis was used to determine if there were overall effects, then univariate analyses. For disciplinary data, independent tests were conducted. In all cases, tests for interactions of condition (program or not) with the covariates were employed.

Outcomes

Multivariate tests of condition on achievement using one-tail tests of significance, controlling for percent free/reduced lunch, student mobility and percent African American students yielded significantly higher results for the PA schools in Nevada, compared to the matched controls. The three covariates were all significant. In addition, univariate tests on school achievement showed significantly higher results for the PA schools compared to their match controls on the following measures: math, reading, and language. For comparisons with all non-PA schools, condition was not significant in the multivariate analysis. For violence data in Nevada, significant program effects were observed for comparisons with all schools and the comparisons with the matched controls for student-to-student and student-to-staff violence and for total number of incidents and incidents per 1,000 students. Marginally significant effects favoring the treatment schools were observed for possession of weapons in the matched control comparison. There were no significant results regarding rates of absenteeism.

In Hawaii, three covariates were significant predictors in the multivariate ANOVA for achievement when comparing PA schools with all other schools: parent education, percent free/reduced lunch, and percent Japanese/Chinese. Condition was also significant and univariate tests showed significantly higher scores in math, reading, and a combined score that was significantly higher for the PA schools, compared to all others. When comparing PA schools with matched controls, the results were parallel with the exception of parent education. For disciplinary data in Hawaii, all indicators (e.g., felonies, misdemeanors, department rules, school rules, and total incidents) were significantly different when PA schools were compared to non-PA schools, and all but misdemeanors were significant when compared with matching controls. Number and rates of absenteeism were also lower for PA schools, when compared to matching controls and all non-PA schools.

Study 4

Flay and Allred (2003) used a quasi-experimental design based on retrospective school-level archival data and matching. The sample included 45 elementary schools in the Southeast that had already implemented the program, 28 controls schools that had never used the program, and 20 control schools that had used other socioemotional learning programs. The outcome measures of student performance and disciplinary referrals/actions came from archival data.

Flay and Allred (2003) found that students in the intervention schools compared to students in the matched control schools showed significantly:

• Higher reading and general achievement
• Fewer violent incidents and suspensions

Evaluation Methodology

Design: This study used retrospective school-level archival data to employ a matched-control research design. The study selected one large southeastern school district that had school-level archival data on student performance and disciplinary referrals/actions easily available for both elementary and secondary schools and that had a significant number of schools that had implemented PA for 4 or more years. Some schools in the district had never used PA or had stopped using it 4 or more years prior to 1997-98 school year (non-PA, n=28). Others had used it for 4 or more years prior to 1998 (PA-only, n=45), and others had also adopted other supplementary character/behavior programs, in addition to continued use of PA (PA+Other, n=20). Each of the latter two groups had used PA for an average of 7 years (range = 4-9 years). These three groups of schools were compared to assess program effects on elementary school student achievement and behavior.

School report card (SRC) data were used to find matching sets of one PA-only school, one PA+Other school and one non-PA control school. Schools were matched by size, percent free and reduced lunch, percent mobility, and ethnic distribution. There were no significant differences between PA-only schools and PA+Other schools. PA schools were not different from the matched control (non-PA) schools on any matching variables. (Retrospective analysis also determined that they were comparable on one of the outcome variables -- academic test scores - thus making this study of stronger design than Study 3 above.) However, PA schools were substantially different than all non-PA schools. PA schools were at lower risk because they had lower proportions of students receiving free/reduced lunch, lower mobility rates, and lower proportions of minority students, but they were at higher risk because they were larger and had a higher student-teacher ratio. Since the evaluators found no significant differences between the PA-only and PA+Other schools on outcomes, these two conditions were combined for the analyses.

For analyses of the sustained effect of PA into middle schools, the proportion of feeder elementary schools that had implemented PA for at least the prior 4 years was calculated. For analyses of sustained effects of PA into high school, the proportion of feeder schools that had implemented PA for at least the 8 years prior was calculated. In each case, the evaluators tried to ensure that students in the middle or high schools would have received at least two years of PA. Each of the 33 middle schools in the district was categorized as low PA (less than 60% of their students being PA graduates), medium PA (60-79% PA graduates), and high PA (80%-100% PA graduates). The 18 high schools in the district were also categorized based on the percentage of their students having graduated from PA during elementary school: low PA (0-15% PA graduates), medium PA (16%-26% PA graduates) and high PA (27-50% PA graduates). None of the high schools had more than 50% of their students coming from elementary schools with PA.

Sample: The matched PA and controls schools averaged 62-68% students receiving free or reduced-price lunch, and had 45-51% white students, 25-28% African American students, and 21-23% Hispanic students. About 37% of the students were reading above the median on normed tests, about 16% were writing above the median, and 45% were performing math above the median.

Measures: Elementary achievement data consisted of mean scores on the Florida Reading Test and the Grade 4 Florida Comprehensive Aptitude Test (FCAT) for the 1997-98 school year. Behavioral data consisted of disciplinary referrals for incidents of violence per 100 students, percent of students who received out-of-school suspensions, and percent of students absent for 21 or more days during the school year.

Middle-school standardized achievement test data were the percent of students scoring above the median on the 8th grade norm referenced tests (NRT) of reading and math (1997-1998). Indicators of behavior included incidents per 100 students of substance use, violence, dissing behaviors (disrespect, disobedience, disorderly, and disruptive), and property crimes. All behavioral data were coded disciplinary referrals by school principals or disciplinary officers. Absenteeism data were also available.

High school standardized achievement test data (1997-98) were the percent of 10th grade students scoring 3 or greater on the Florida Writes test, percent of seniors passing the High School Competency Tests (HSCT) of communications and math, mean Scholastic Aptitude Test (SAT) scores, and mean American College Testing (ACT) composite scores. Percent absent 21 or more days and percent dropout were other indicators of school involvement. Behavioral data (1998-99) included disciplinary referrals for substance use (tobacco, alcohol, and illicit drugs), violence (threatening, fighting, carrying weapons, and battery), dissing behaviors, sexual behaviors, property crime, breaking of school rules, misbehavior on or near school buses, parking violations, and falsification of reports. Data on percent of students suspended (separately for in-school and out-of-school) was also included.

Analysis: Multivariate general linear modeling (GLM) with fixed effects for condition and pair number was conducted for the comparison of matched PA and non-PA schools. To estimate the effects of receiving PA in elementary school on achievement and behavior in middle and high school, the evaluators conducted GLM for each set of outcomes using percent PA students as the independent variable and using percent free/reduced lunch, school size, and percent mobility as covariates.

Outcomes Posttest and Long-Term:

Elementary School Results - Scores on the reading test and FCAT were significantly higher for schools receiving PA in both the all-schools analysis and for those in the matched control analysis. The number of violent incidents per 100 students was significantly lower in PA schools than comparison schools in the all-schools analysis, and in the matched-school analysis. The percentage of students receiving out-of-school suspensions was significantly lower for the matched-schools analyses and marginally lower for the all-school analyses. No significant effects were reported for absenteeism.

Middle School Results - For all outcomes, middle schools with more PA graduates scored significantly higher than schools with fewer PA graduates. These results included reading scores, math scores, behavioral incidents per 100 students for drug use, violence, disorderly conduct, and property crime.

High School Results - Significant outcomes suggesting favorable program results were reported for the following indicators: percent scoring greater than 3 on Florida Writes, percent passing HSCT tests of communications, mean SAT scores, percent continuing education, percent school drop-out, substance use, violence, sexual behavioral problems, dissing behaviors, absent over 21 days, and percent in- and out-of-school suspensions. No significant effects occurred for behaviors related to property crime, school rules, busing, and parking. For all significant outcomes, there was a clear dose-response relationship with higher dosage schools experiencing better outcomes.

Outcomes - Brief bullets:

• Scores on the reading test and FCAT were significantly higher for elementary schools receiving PA in both the all-schools analysis and the matched control analysis.
• The number of violent incidents per 100 students was significantly lower in PA elementary schools than comparison schools in the all-schools analysis and in the matched-schools analysis.
• The percentage of students receiving out-of-school suspensions was significantly lower for PA schools in the matched-schools analysis and marginally lower for PA schools in the all-schools analysis.
• For all outcomes, middle school students with more PA graduates scored significantly better than schools with fewer PA graduates. These results included reading scores, math scores, behavioral incidents per 100 students for drug use, violence, disorderly conduct, and property crime.
• Significant outcomes suggesting favorable program results were reported for the following indicators for high schools with a higher percentage of students that received PA: percent scoring greater than 3 on Florida Writes, percent passing HSCT tests of communications, mean SAT scores, percent continuing education, percent school drop-out, substance use, violence, sexual behavioral problems, dissing behaviors, absent over 21 days, and percent in- and out-of-school suspensions.

Generalizability: Since probability-sampling techniques were not used to select the PA program schools, it is unclear whether these schools are representative of elementary schools in the district in which they reside or of elementary schools in general.

Study 5

Summary

Washburn et al. (2011) used a cluster randomized control trial with eight rural public schools. The schools were randomized in even numbers to intervention and control groups and included 5,100 children in grades K-4. Trajectories in positive behavior were studied over a three-year period.

Washburn et al. (2011) found that students in the intervention schools compared to students in the control schools scored significantly higher on:

• Endorsement of prosocial behaviors

Southeastern State Trial (Washburn et al., 2011)

Eight rural public elementary schools, with five age cohorts ranging from kindergarten to grade 4, were matched and randomized (no details available regarding randomization). Seven school-level variables were available at baseline which indicated no differences between treatment and control schools. There were 1,652 students in the first wave, 1,944 in the second, and 1,504 students at the third wave. The frequency of positive behaviors associated with character was asked at the end of the first through third years; this is the only outcome measure. There was no pretest. The trajectories of children were compared from the first year of intervention through the end of the third year of intervention using a multi-level, growth-curve analysis. There was a significant intervention effect with Positive Action mitigating the decline in the endorsement of positive behaviors by students.

Study 6

This study evaluated an extension of the kindergarten curriculum to preschool-aged children, but for the initial implementation, it included only the school-based component of the program, without the community component. Also, the study examined the impact of the program after completion of 60 of the 130 lessons.

Summary

In a study of the pre-school program, Flay (2012) and Schmitt et al. (2014) used a quasi-experimental design with 12 instructors non-randomly assigned to intervention and control groups and 135 students randomly assigned to the instructors. The posttest assessment after 3-4 months of the program examined a variety of teacher-rated child outcomes such as physical health, self-concept, self-control, and social bonding.

Flay (2012) and Schmitt et al. (2014) reported significantly higher instructor ratings at posttest for the intervention children than control children on outcome measures relating to:

• Physical health
• Self-concept
• Self-control
• Social bonding

Evaluation Methodology

Design:

Recruitment: The convenience sample came from three preschools in Virginia during the fall of 2009. Instructors from 12 classrooms agreed to participate in the study. With an average of 15 children per classroom, the total number of children would be 180.

Assignment: In a quasi-experimental design, teachers self-selected into the intervention group (n = 7) or business-as-usual control group (n = 5). Although the choices were said to be based primarily on scheduling, self-selection could confound teacher motivation with the program. However, children were randomly assigned to a classroom instructor.

Assessments/Attrition: The pretest in September was followed by a posttest in December- January. At pretest, 12 instructors from 3 sites rated 146 students, while at posttest, 11 instructors from 2 sites rated 163 students. The study analyzed those students who completed both pretest and posttest, leaving 135 students (92%) from 11 instructors.

Sample Characteristics: The study lacked demographic information, stating only that 54% of the children were male.

Measures:

Instructors both delivered the program and provided all child outcome measures. The instructor ratings covered 11 domains addressed by the Positive Action program plus a combined total measure: understanding PA, self-concept, physical health, intellectual health, self-management, self-control, respect, consideration, social bonding, honesty, self-improvement, and total. The 11 domain scales had alpha reliability coefficients between .76 and .93, and the total scale had an alpha of .98.

Analysis: ANCOVA models adjusted for pretest scores but not for clustering within classrooms, the unit of assignment.

Intent-to-Treat: The study used data from all students who completed both the pretest and posttest assessments.

Outcomes

Implementation Fidelity:

Four of six instructors reported that they delivered nearly all the lessons. Students received an average of 4.8 lessons per week over 10 weeks, about 48 of the 60 lessons.

Baseline equivalence:

Intervention children had significantly higher pretest means on 10 of the 12 outcomes.

Differential attrition:

No analysis of missing data or attrition was presented.

Posttest:

Intervention teachers rated the children as significantly higher than control teachers on all 12 outcomes. The effect size for the total score was 0.62, and the effect sizes for the domain measures ranged from 0.36 (self-management) to 0.72 (self-control).

Long-term: Not available.

Study 7

Summary

Guo et al. (2015), Smokowski et al. (2016), and Stalker et al. (2018) used a quasi-experimental design that examined two rural, economically disadvantaged counties in North Carolina, one with middle schools that received the intervention for three years and one with schools that did not. To adjust for non-random county assignment, the analysis used propensity score matching of students. The sample included more than 4,000 students in grades 6-8 from 27 middle schools and 11 high schools. Data obtained over three years included outcome measures of self-esteem, school hassles, aggression, and internalizing symptoms.

Compared to the control group in an independent evaluation (Guo et al., 2015), students in the intervention condition showed:

• Improvements in self-esteem and school hassles scores
• Possible iatrogenic effect on internalizing symptoms

For those receiving three years of the intervention and a high number of PA lessons, compared to matched control youth (Smokowski et al., 2016), the intervention group reported significantly higher

• Self-esteem

Evaluation Methodology

Design:

Recruitment: More than 4,000 students from 27 middle schools and 11 high schools in two rural, economically disadvantaged counties in North Carolina were included in the North Carolina Academic Center for Excellence in Youth Violence Prevention project. The Guo et al. (2015) and Smokowski et al. (2016) studies used the same data from 4 waves of the Rural Adaptation Project panel data collected between 2011 and 2014. The sample size, depending on year and analysis, ranged from N=1246 to N=5894. Stalker et al. (2018) reported a sample size of 8,333, using an additional wave of data in year 5.

Assignment: The data came from two counties non-randomly assigned to conditions. The control county sampled all middle school students (6th - 8th grade) in Year One of the study, and all incoming 6th grade students in the subsequent years. The intervention county was larger, and although all students in the county received the intervention, 40% of all middle school students (grades 6-8) were randomly sampled for the first wave, and 500 more incoming 6th grade students were randomly sampled in every subsequent year. The intervention was delivered for three consecutive years. To adjust for non-random county assignment, the analysis used propensity score matching. Guo et al. (2015) compared matched intervention and control groups, while Smolenski et al. (2015) compared matched groups defined by dosage levels. Stalker et al. (2018) did not use any matching protocol and focused on dosage only, studying years of implementation rather than comparing treatment and control schools.

The study noted that consent was requested from parents in the intervention schools but apparently not in the control schools. Stalker et al. (2018) reported that parents were asked to opt out of participation.

Attrition: Data were collected in four waves and the sample size ranged overall from 3715 to 5894 and from 1246 to 1968 for propensity score matching. Other than to say that subjects needed at least two waves of data to be included in the analysis, the study did not provide details on attrition. Stalker et al. (2018) reported an additional wave of data. Attrition between Waves 4 and 5 was 23%, however no additional information was provided on attrition.

Sample:

The sample was 50-52% female with an average age of 12.78. A majority of the sample received free or reduced price lunch (79-88%) and lived in a two-parent family (70-92%). The racial/ethnic breakdown of the sample was 25-27% White, 23-25% Black, 25-30% American Indian, 8% Latino, and 12% identified as mixed or "other."

Measures:

Participants were assessed using the School Success Profile Plus, which includes numerous subscales, but only four were used as outcomes in the analysis in Guo et al. (2015) and Smokowski et al. (2016): self-esteem, aggression, internalizing symptoms, and school hassles. Other measures such as friend rejection, parent-child conflict, religious orientations, school satisfaction, future optimism, parent support, teacher support, friend support, delinquent friends, peer pressure, perceived discrimination, and school danger were used in the matching. The alphas for all the measures ranged over time points from .70 to .97. For the primary outcomes of self-esteem, school hassles, aggression, and internalizing, the alphas ranged over time from .86 to .95.

Stalker et al. (2018) reported using most of the original scales plus 12 additional scales. The researchers reported results for subscales of school hassles - including verbal, physical, and relational victimization - as a mediator and alcohol use, aggression, depression, and anxiety as outcomes. The study reported high measures of reliability for most measures (Cronbach's alpha .84-.94).

Analysis:

Guo et al. (2015) and Smokowski et al. (2016) matched conditions using two propensity score models (inverse weighting and one-to-one matching) that accounted for the difficulties with non-random assignment by county. Then hierarchical linear modeling was used to analyze individual change over time with controls for baseline covariates used in the matching. Tests for clustering showed an intraclass correlation coefficient of only .03, which led the authors to ignore school clustering in the hierarchical models. Also, the tests for program effects did not examine differences across conditions in change with a time-by-condition term, but appear to examine the average condition difference across all assessments with a condition term alone.

Stalker et al. (2018) did not match conditions. Outcome measures were analyzed with structural equation modeling to examine latent variables and test for effects of program dosage (0 for control schools and 0-3 years for treatment schools). The analysis used data only from year 4 (the last year of the program) and year 5 (1-year follow-up). A structural model was specified to examine whether there was an indirect relationship between dosage and adolescent outcomes through school hassles. The following paths were tested: 1) the effect of dosage on school hassles, 2) the effect of school hassles on adolescent outcomes, and 3) the direct effect of dosage on adolescent outcomes. Full information maximum likelihood was used to account for missing data.

Intent-to-Treat: The study imputed missing data for subjects with data for at least two assessments. Based on data using multiple imputation, the tables in Guo et al. (2015) listed 1) average treatment effects for an intent-to-treat sample, 2) average treatment effects of the treated for a non-intent-to-treat sample, and 3) effects for a matched subsample of subjects. The sample sizes were not presented clearly, but the study appears to have followed intent-to-treat procedures by using as many subjects as possible in the average treatment effects analysis. Also using multiple imputation, Smokowski et al. (2016) presented results for the full sample of subjects with at least two assessments.

Stalker et al. (2018) used all available data, dropping only those missing all outcome data.

Outcomes

Implementation Fidelity:

Teachers were trained on program implementation, but fidelity was assessed via program dosage, or the number and duration of lessons taught. The authors concluded that fidelity was high, particularly in years 2 and 3.

Baseline Equivalence:

Conditions differed significantly on most baseline measures for the original sample, but Guo et al. (2015) stated that the matched samples were balanced and that details could be requested. In Smokowski et al. (2016), equivalence was tested across dosage groups rather than conditions. They noted that 10-11 variables were not balanced after the propensity score adjustment. Stalker et al. (2018) did not match conditions and reported that the implementation and control counties differed in some ways, including racial make up, median income, and rates of youth violence (p. 149).

Differential Attrition:

Perhaps because of the use of multiple imputation, details on attrition or an attrition analysis were not provided. Stalker et al. (2018) did not use multiple imputation but also did not provide details on differential attrition.

Posttest:

As reported by Guo et al. (2015), the program had a significantly positive effect on scores for self-esteem and school hassles in the intervention group compared to the control group. It did not significantly affect aggression, and it may have had a negative effect on internalizing scores (the authors stated that the effect was significant in a two-tailed test but not in the one-tailed).

Using the same data, but reporting one-tailed significance tests, Smokowski at al. (2016) found 3 significant dosage effects in 28 tests. Students who received 3 years of the PA intervention and a high number of PA lessons had a significantly higher self-esteem score than those who received 0 years of PA or zero lessons (p < .01). Participants who received one year of PA also reported lower school hassle scores than those who received 0 years, but at p < .05, this may not have been significant with a two-tailed test. Dosage was not related to aggression. One-tailed tests of significance showed no relationship between dosage and internalizing symptoms, but the results were in the opposite direction, showed relatively large coefficients, and may have been significant with 2-tailed tests. Specifically, youth who received 3 years of PA, and those who received a low (0-31) and moderate (63-103) number of lessons, had higher internalizing scores compared to youth who received 0 years of PA or zero lessons.

Long-Term :

The data were collected in four waves, but the program was ongoing over the full period in Guo et al. (2015) and Smokowski et al (2016).

Stalker et al. (2018) followed subjects for a year after then end of the 4-year program. They reported no significant direct or total effects, with one iatrogenic exception. The total effect of program dosage on alcohol use, which combined the direct and indirect effects, was significant and positive. Aggression, anxiety, alcohol use, and depression showed significant indirect effects through school hassles.

Study 8

The study evaluated a version of Positive Action that was integrated with the existing Health and Family Life Education curriculum in Belize. Teachers and other school staff were allowed flexibility to adapt lessons and other program activities for purposes such as cultural appropriateness, fit with the school environment, and integration with content from the Health and Family Life Education curriculum.

Summary

Hull et al. (2021) used a cluster randomized trial to examine 4,575 students in grades 1-6 (ages 7-12) across 24 primary schools in Belize City, Belize. Students in the randomly assigned schools were assessed at baseline (the start of the school year) and posttest (the end of the school) to measure components of positive youth development.

Hull et al. (2021) found one significant intervention effect in tests for 16 outcomes. The intervention group reported significantly higher scores than the control group on:

• A scale of positive engagement behavior in grades 1-3

Evaluation Methodology

Design:

Recruitment: The sample was drawn from 6,296 students in grades 1-6 (ages 7-12) across 24 primary schools in Belize City. The schools were selected randomly from the Belize District. The authors provided no information on consent rates but stated the overall student participant sample was 4,575.

Assignment: Schools were randomly assigned to treatment and control groups within strata defined by mean cognitive ability, urban/rural status, and school size. The 12 control schools included 2,358 students, and the 12 treatment schools included 2,217 students. The control schools received a separate teacher training program unrelated to the social-emotional/character development of Positive Action and an alternative intervention focused on mathematics instruction and the use of manipulatives.

Assessments/Attrition: Assessments came at the beginning (baseline) and end (posttest) of the 2011-2012 school year. The authors presented no information on attrition or missing data.

Sample:

The sample consisted of 43% males, 49% females, and 8% unreported. Ethnicity had the following distribution: Creole (50%), Garifuna (9%), Maya (3%), Mestizo (27%), and Other and not reported (11%).

Measures:

The measures came from student surveys that differed for standards or grades 1-3 (ages 7-9) and standards or grades 4-6 (ages 10-12). The three self-reported measures for younger students reflected engagement in positive behavior, feelings about positive behavior, and a combined scale of positive youth development. The 12 self-reported measures for the older students covered a variety of behaviors (anxiety, substance use and violence, self-control) and risk and protective factors (rewards for prosocial behavior, neighborhood participation, peer affiliation, honesty, respect for teachers and parents, self-development, moral beliefs, social-emotional character development, prosocial behaviors). Omega reliability coefficients for the measures, with few exceptions, reached or exceeded .70.

Analysis:

The analyses used hierarchical linear models with an unrestricted covariance structure to adjust for clustering. However, school-level effects were aggregated to the teacher (classroom) level rather than the school level, the unit of randomization. The models include pre-treatment scores for the outcomes. All models used multiple imputation, though the authors offered no details on the extent of missing data or the imputation model.   

Intent-to-Treat: The analysis removed students if they were not assigned to a teacher or did not participate in either pre- or post-treatment assessments. All other students appear to have been included with multiple imputation, but the lack of information on attrition makes it uncertain.

Outcomes

Implementation Fidelity:

Not examined.

Baseline Equivalence:

The study noted only that the randomization produced balance on the three covariates that were used to define the randomization strata (cognitive ability, urban/rural status, and school size).

Differential Attrition:

The authors stated that "data was evaluated using Little's MCAR test and determined the data to be to be missing at random." This appears to mean that the Little test rejected the null hypothesis of missing completely at random, but it is not possible for the test to show data are missing at random rather than missing not at random. No other information was provided.

Posttest:

As shown by the treatment indicator g₀₁ in Table 3, the tests indicated that one of 15 outcomes showed a significant condition difference at posttest: the intervention group compared to the control group reported a higher score on engaging in positive behavior in grades 1-3. The authors noted that, despite the lack of significant individual effects for the 12 outcomes measured for grades 4-6, the joint multivariate test of the treatment for all 12 outcomes was significant.

Long-Term:

Not examined.