Olweus Bullying Prevention Program

Study 1

Evaluation Methodology

Design: The Olweus Bullying Prevention Program in Norway was the result of a state mandated effort to combat serious bullying in Norwegian schools and classrooms. The catalyst for the mandate was the suicide deaths of three Norwegian school children with a history of victimization. The gravity of the tragedies, the media attention and government financing and coordination of the program fostered a great deal of support by teachers, parents and the general public.

Because the Bullying Prevention Program was a nationwide anti-bullying program, random selection of treatment schools was not possible. The evaluation was a longitudinal, quasi-experimental design with time-lagged contrasts of four adjacent grade cohorts. In this design, the grade 5 cohort at T1 was compared with the T2 data for the grade 4 cohort, which at that time had reached the same age as the baseline group. The same kind of comparisons were made between the grade 6 cohort at T1 and the grade 5 cohort at T2 and between the grade 7 cohort at T1 and the grade 6 cohort at T2. Only two of the cohorts could be used to assess T1 and T3 effects. Grade 6 and grade 7 cohorts were contrasted with data collected at T3 on the grade 4 and grade 5 cohorts, respectively.

All participants were administered a Bully/Victim questionnaire, assessing the nature and prevalence of bullying in their school, in the spring of the school year prior to program implementation (May 1983). Follow-up questionnaires were administered in the spring of the following two school years, i.e. after eight months of treatment (May 1984) and twenty months of treatment (May 1985).

Sample: The sample included 2,500 5th-8th grade students in 112 classrooms in 42 schools in Bergen, Norway. Each cohort consisted of between 500 and 700 students. Classes were the sampling unit used in analyses.

Measures: Measures taken from the Bully/Victim Questionnaire included (1) items on the questionnaire regarding the frequency of bullying incidents during the semester, (2) students' attitudes toward bullying and (3) teacher responses to bullying. Students were also given a 23 item questionnaire regarding self-reported anti-social behaviors, items assessing school climate. Additional measurements included teacher ratings of classroom bullying behaviors and a four-dimensional measure of classroom climate.

Analysis: Analysis of variance (ANOVA) was used to capture the effects of students within classes within schools within cohorts. To determine the intervention effects controlling for the cohort effects, comparisons were made for age/grade equivalent cohorts. In other words, separate analyses for each combination of age/grade cohorts were conducted comparing Time 1 (T1) to Time 2 (T2) and Time 1 to Time 3 (T3).

Outcomes

Bergen, Norway (Olweus & Alsaker, 1991):
Students' reports of bullying and victimization were significantly reduced, in most cases by 50% or more. These reductions were generally consistent across gender and grade subgroups. Some program effects were significantly stronger at the second follow-up (twenty months post baseline) compared to the first follow-up (eight months post baseline). Teacher and peer reports also indicated a decrease in bully/victim problems. Students reported an improvement in order and discipline, positive social relationships and attitudes toward school, while also indicating reductions in anti-social behavior, such as theft, vandalism and truancy, all of which indicate an improvement in overall school climate. Students reported an increase in their satisfaction with school.

Additional analyses indicated a cohort effect in bully/victim problems among the four grades included in the evaluation. However, when the cohort effect was controlled for, the intervention effect from baseline to follow-ups remained significant. Also, classes that received stronger implementation of integral program components showed greater reductions in bully/victim behaviors.

Study 2

Evaluation Methodology

Design: The evaluation was a quasi-experimental design with non-random assignment of treatment and comparison sites in matched pairs. The evaluation sample was comprised of 4th, 5th and 6th grade students in six non metropolitan districts in South Carolina. The districts were divided and matched into pairs based on location and student demographics. Districts were predominately African-American, poor and located in counties with high juvenile delinquency rates. Baseline data was collected in 1995 (Y1), when participants were in grades 4-6. Follow-up data was collected in March of 1996 (Y2) and 1997 (Y3). All schools in one district in each pair received the program in Y1, after baseline data collection. These schools formed group A. Schools in the other district in each pair received the program one year later, at Y2. These schools formed group B. Group B served as a control group for Y1. There were 11 group A schools in Y1 and 28 group B schools. In Y2, all group A schools continued in the evaluation, and 7 group B schools began the program. Data was collected from 6,389 students in Y1, 6,263 students in Y2 and 4,928 in Y3.

Sample Characteristics: Data was collected from 6,389 students in Y1, when students were in grades 4-6. In Y2, when the sample was in grades 5-7, 6,263 students were surveyed. In Y3, when students were in grades 6-8, there were 4,928 students in the sample. Schools in the sample were predominately African American (ranging from 46-95%), poor and located in counties with high juvenile arrests rates in the state (top 15% of all counties in S.C.).

Measures: The evaluation used the original Bully/Victim Questionnaire developed by Olweus for the Norway program (see primary study). Of the twelve scales, there were measures of bullying frequency, victimization frequency, frequency of bullying of teachers, and attitudes about bullying. Eight scales measured self-reported anti-social behavior (i.e. theft, vandalism and truancy), violence, delinquency, school misbehavior, school sanctions, group delinquency and substance use.

Analyses: Hierarchical linear modeling (HLM) was used to estimate program effects. Gender and grade were included as Level 1 variables, as was overall program impact. This variable compares scores from baseline to scores after each group has experienced one year of the intervention. Also at Level 1, the time variable was coded to distinguish the effect of the second year of the program (time 3). A Level 2 variable was included to provide a test of group differences at each point of measurement (group). Analysis of covariance (ANCOVA) was used to determine group differences after one year of the program, at T2.

Outcomes
Baseline Equivalence and Differential Attrition: Group A had higher rates bullying peers and bullying teachers, but Group A was equivalent to Group B on measures of victimization and attitudes toward bullying.

Posttest: Results of HLM demonstrated no significant differences between Group A and Group B in bullying peers, victimization, anti-social behavior, attitudes toward bullying or bullying teachers. There were no overall program effects (combining the effects for both years for group A and the second year group B) or year 2 effects.

ANCOVA, used to determine group differences after one year of the program, indicated that Group A after one year of the program, compared to Group B, had reduced rates of bullying and anti-social behaviors, i.e. theft, vandalism and truancy, but no differences on measures of victimization and bullying teachers.

HLM analyses indicated the program demonstrated positive results on measures of bullying and anti-social behavior, i.e. theft, vandalism and truancy, for group A after one year of treatment, but these differences were not sustained, as indicated by follow-up surveys in Y3. Group B, who received delayed program implementation, demonstrated no program effect.

Study 3

Evaluation Methodology

Design: The evaluation was a quasi-experimental design with adjacent age-cohorts. Baseline data was collected in November of 1990 and schools began program implementation in 1991. The first steps of program implementation, inviting schools to participate in the program, began in February of 1991. Teacher training workshops took place in spring and summer and program implementation and monitoring began in September of 1991. Follow-up data was collected in November of 1992, after a full year of treatment.

The participants were 8 to 11 years old (enrolled in junior/middle schools, also referred to in the article as primary schools) and 11 to 16 years old (enrolled in secondary schools). A total of 6,758 students from twenty four schools participated in surveying at baseline (2,623 from 17 primary schools and 4,135 from 7 secondary schools). One primary school and three secondary schools declined program participation, but agreed to be surveyed at follow-up. These four schools serve as comparison sites, who did little or nothing to address the problem of bullying.

Implementation and program development did not proceed along the optimal timeline. At the time of the follow-up survey only 8 primary schools and 4 secondary schools had made the expected progress through stages of development. In order to ensure comparability between baseline and follow-up, researchers omitted certain classes from each year so that the numbers of classes and pupils were equal at both survey times.

Sample Characteristics: The baseline sample was comprised of 6,758 students (1,271 boys and 1,352 girls) from 24 schools. This included 17 primary schools, with 112 classes of 2,623 students ages 8-11 and 7 secondary schools, with 211 classes of 4,135 students ages 11-16. The schools represented all areas of the city.

Measures: The student self-report survey was a modified version of the original Olweus Bully/Victim Questionnaire. At the follow-up survey, students were asked if they perceived the school had taken any action to stop bullying and if there had been any change as a result of that action.

Analyses: Change in means between the first year (T1) and second year (T2) surveys were compared using single-sample t-tests and probabilities for percentage difference scores.

Outcomes
Baseline Equivalence and Differential Attrition: After the researchers made adjustments to ensure comparability between baseline and follow-up, there were 2,212 primary school students and 4,256 secondary school students in treatment schools, and 99 primary students and 1,742 secondary students in comparison schools. The total number of students aged 8-16 years was 8,309.

Posttest: Most individual treatment schools indicated increases in the number of students who did not bully others, a decrease in the frequency of bullying others and a decrease in the number of peers who bully others. The only significant result across all schools was a decrease in students' self-reported frequency of bullying other students (a 12% decrease). In most schools, there was an increase in the number of students who reported spending break time alone. There was also an increase in the number of students who indicated they would not join in bullying. A larger percentage of students were reporting to adults that they were being bullied and that an adult had talked to them if they had bullied others. However, there were no differences in their perception of teachers stopping bullying. For most measures, changes were better in treatment schools than in comparison schools. Among comparison schools, one primary school and one secondary school outcome measure indicated bullying problems had worsened between baseline and follow-up.

Long-Term (Eslea, M., 1998).
This study is a follow-up study conducted in late 1993 after 2 years of intervention in the Sheffield evaluation. The follow-up contained 4 schools from the original sample and was conducted one year after the second survey (the last survey) of the Sheffield students. Of the original primary schools, 11 head teachers agreed to interviews, and four of the schools subsequently took part in a third survey. The surveys were done on a whole-school basis, with 657 children, aged 7-11, participating. Surveys were administered at the same time of the year as those in the earlier Sheffield evaluation. The results for these four schools were compared with their previous results.

Two of the schools showed a decline in students who reported being bullied sometimes or more since the start of the project. One school experienced increases over time in students who reported being bullied, and one school experienced a decline initially but then experienced an increase. Overall, boys experienced a decline in being bullied over time, while girls first experienced a decline and then an increase. By the third survey, girls were significantly more likely to report being bullied than were boys. The results were similar with regard to bullying others, with two schools showing a decline, one school an increase and one school a decrease and then an increase. After two years of intervention, there was little to no change in the percentage of students who told someone at home or at school about bullying. There were also no overall differences between 1992 and 1993 of the anti-bullying effort made by the schools and the effectiveness of that work as perceived by students, although individual schools did show some relationship between effort and effectiveness.

Study 4

Evaluation Methodology

Design: The evaluation was a pre-test post-test design. Schools participating in the program and evaluation were self-selected due to their interest in and commitment to bullying prevention efforts. A total of 1,052 students from four treatment schools (three K-8 schools and one grades 7-8) participated in the evaluation at baseline. Eleven fewer students (1041) participated in the follow-up survey. Baseline surveys were administered before program implementation, and follow-up surveys were administered 18 months post baseline. Supplementally, teachers were asked to complete a qualitative survey on classroom activities. Seventy-four percent completed surveys. Eight team leaders were interviewed.

Sample Characteristics: The sample included 1,052 students from four treatment schools (three K-8 schools and one grades 7-8). All schools were in urban areas with diverse ethnic populations. All students in grade 3-8 in all four schools were surveyed. Eleven fewer students (1,041) completed the follow-up questionnaire due to school mobility, baseline and follow-up samples were not paired. Seventy-eight teachers (74%) completed a Classroom Activities Questionnaire and eight team leaders participated in interviews.

Measures: The 40-item student self-report questionnaire on bullying and victimization was closely based on the original Bully/Victim Questionnaire developed by Olweus (described in primary study). The reference period was two months.

The Classroom Activities Questionnaire asked teachers about implementation of program components in their classrooms. The team leader interviews were based on school leaders' perception of program implementation and success indicated by observed change in attitudes and behaviors among students and staff.

Analyses: For the student bully/victim questionnaire, the differences in frequencies of behaviors between baseline and follow-up were calculated. A z -test was used to determine significant differences in the distribution of responses between baseline and follow-up. Teacher Classroom Activities Questionnaire and school leader interview data were qualitatively analyzed.

Outcomes
Posttest: According to students self-reports, there were significant increases in teacher interventions and teachers talking to bullies about their behavior. There was no difference in the frequency of victims talking to teachers, nor was there change in parents talking to their children. Students reported being less likely to engage in bullying (17% reduction). The number of children reporting that they or their peers had intervened in bullying and those who reported feeling uncomfortable observing bullying did not change. There was a small but significant increase in the percent of children who self-reported they had bullied other students (2% increase over the school term, and 5% increase in the past five days). A smaller percent reported being victimized in the past five days (18% decrease), but there was no change over the school term. Racially motivated bullying increased. More children reported spending recess alone.

Results were mixed. Bullying behaviors increased over time, although students reported they were less likely to join in bullying behaviors. Victimization decreased on one measure, but not another. Teachers, but not parents, were more likely to talk to bullies about their behavior, but neither were more likely to speak to victims about their difficulties.

Study 5

Evaluation Methodology

Design: This program implementation was evaluated using a pre-post design. Of the 100 schools that were invited to participate, 42 (42%) accepted. Third and fourth class (or 3rd and 4th grade) students completed pre-test assessments at the beginning of the school year and posttest assessment at the beginning of their next school year. Only 22 schools were included in the analysis (52% of those involved, 22% of those invited) because of problems in the remaining schools with administration of the questionnaires to the correct class groupings. Only 22 schools had matching pretest (n = 527) and posttest (n = 520) data. A teacher questionnaire was also implemented to 126 teachers at pretest and 83 at posttest (66% retention rate).

Sample Characteristics: This report provides no specific information on the composition of the sample.

Measures: The student questionnaire was a modified version of the Olweus Bully/Victim Questionnaire. The teacher questionnaire assessed teacher's awareness of, and response to, bullying.

Analysis: Results were generated through chi-square tests with post-hoc analyses implemented to assess the effect of age on program effects.

Outcomes
Baseline Equivalence and Differential Equivalence: There were no differences between student reports at pre- or post-test based on age. The two grades of students were similar on their experiences being bullied and their experiences bullying others. The pre- and post-test teacher samples were also closely matched on gender, age, experience teaching, and employment positions held.

Posttest:
Student Reports: There were significant reductions in self-reported victimization. The percentage of students who were bullied once a week or more (7.2%) significantly declined (to 3.6%), as did the percentage of those who reported being bullied in the past five days. There was only a marginally significant decline in self-reported incidents of bullying in the past school term, but those who reported bullying others once a week or more in the past school term significantly declined from pre- to posttest, as did self-reported bullying in the past five days. The program had no effect on the reporting of bullying or on students' perceptions of the willingness of teachers or peers to try to stop bullying. Students were, however, significantly more likely to report that they would try to stop bullying if they saw it occurring. Finally, significantly fewer students reported at posttest that if they saw a student being bullied, they would do nothing.

Teacher Reports: Tests of significance were not performed or were not reported in this publication on the teacher-report data. The majority of teachers felt the school was a safe place for young people who find it hard to defend themselves and claimed that they were interested in stopping bullying, "always" attempted to stop bulling (96% at pretest, 97.6% at posttest) - or usually did, and believed that bullying is an issue for all staff to handle. Support for the notion that more rigorous monitoring of school bullying was needed dropped from 67.5% to 51.8% at posttest.

Study 6

Evaluation Methodology
This evaluation reports on program implementation in 37 Oslo schools who started the program in 2001 or 2002. There were 24 primary schools, 6 lower secondary schools, and 7 combined schools (grades 4 through 10) with a total of about 8,000 students. All schools carried out assessments at pre-test and one year later (after 8 months of the program) and about half of them completed a third assessment, two years after the first (after 20 months of the program). Outcomes included being bullied, defined as being bullying two or three times a month or more, and bullying other students, defined as bullying others two or three times a month or more.

Outcomes
Posttest: Before starting the program, the general prevalence of being bullied was 14% for primary schools and 7% for lower secondary schools. The general prevalence of bullying another student was 6-7% at both primary and lower secondary schools. Eight months after the program began, data from the primary schools and combined schools were combined due to similarity. For these schools, there were significant reductions of 30-40% in the prevalence of being bullied and of 30-45% in the prevalence of bullying others. For lower secondary schools, there were no changes for being bullied and only a 5% reduction in bullying others. Twenty months after program implementation began, reductions continued in all schools, with 30-35% reductions (from baseline) in bullying others in lower secondary schools. In primary schools, there were 60% reductions from baseline in bullying others.

Additional analyses indicate that results for boys and girls were quite similar. The positive results also applied to students involved in bullying that took place once a week or more, students involved in relational bullying (exclusion, ostracism, social isolation), and for bullying in the form of sexual harassment.

Study 7

Evaluation Methodology

Design: This publication reports on a quasi-experimental evaluation of program implementation in Seattle middle schools (grades 6 through 8) in the 2003-2004 academic year. Ten schools were followed after the state of Washington mandated that middle schools implement anti-bullying policies. Schools were given the freedom to implement the program or policy of their choice. Seven schools chose to implement the OBPP, while three utilized less formal activities. Students completed assessments in the spring of 2003 (semester before program implementation) and again in the spring of 2005 (one year after program implementation completed). There were 4,959 intervention students and 1,559 comparison students.

Sample Characteristics: 51% of intervention youth and 50% of comparison youth were female. The intervention group was 12% Black, 7% Hispanic/Latino, 24% Asian, 40% White, 2% Native American, and 9% other, while the comparison group was 28% Black, 7% Hispanic, 25% Asian, 23% White, 1% Native American, and 10% other. Gradewise, 34% and 37% of treatment and comparison youth, respectively, were in the 6th grade, while 33% each were in 7th grade and 32% and 29% respectively were in 8th grade. In each group, 44-47% made mostly As, 47-48% made mostly Bs/Cs, and 6-8% made mostly D/Fs.

Measures: Questions from the Revised Olweus Bully/Victim Questionnaire were used to measure relational and physical bullying victimization (4 items), attitudes and feelings towards bullying (1 item), perceptions of others' readiness to intervene (2 items), perceptions of safety of school (1 item), perceptions of school supports (7 items), and school engagement (1 item).

Analysis: Student-level data were aggregated by school, and school-level data were used in analysis. Poisson regression was performed, controlling for several school-level variables, including school size, percent of students eligible for free/reduced lunch, percent of students meeting state standards for a reading achievement test. Analyses also controlled for baseline frequencies of victimization, age, gender, and ethnicity.

Outcomes
Baseline Equivalence and Differential Attrition: Groups were equivalent at baseline on gender, grade level, achievement, and school-level characteristics. The comparison group had significantly more Black students than intervention schools, while intervention schools had significantly more White students than comparison schools. Baseline frequencies of both relational and physical victimization were lower for the intervention group than the comparison group, but it is not specifically stated whether this difference was statistically significant. There were significant differences between groups, however, on pro-victim attitudes and desire to help others at baseline.

Because interview assessments did not contain personal information, it is not known if the students who completed baseline also completed follow-up assessments. Retention rates, in terms of number of assessments completed at baseline versus those completed at follow-up, varied depending on outcome measure (physical victimization, relational victimization, and attitudes). For youth in intervention schools, these retention rates ranged between 89.1% and 97.7%, while they ranged from 92.9% to over 100% for comparison youth.

Posttest: At one-year posttest (though one of the intervention schools continued program implementation through the academic year after post-test), there were no program effects on relational or physical victimization. When stratified by race, however, White students in intervention schools reported significantly less relational and physical victimization, compared to White students in comparison schools. There were no effects for youths of other races and no effects were seen when examined by gender or grade level. There were also no overall program effects on student attitudes/feelings towards intervening in bullying situations, school engagement, perceptions of school supports, or perceptions of school safety. Sixth grade intervention youth, however, were significantly more likely to feel sorry for bullied students and want to help them, compared to 6th grade youth in comparison schools. Finally, while intervention youth overall were significantly more likely to perceive other students as actively intervening in bullying incidents than comparison youth, there were no program effects on student perceptions of teachers' or other adults' readiness to intervene.

Study 8

Evaluation Methodology

Design: The study utilized a quasi-experimental design with time-lagged contrasts between age-equivalent groups. Students who received one year of the program were compared with children of the same grade level who did not receive the program. A cohort of 158 7th and 8th grade students in one small Catholic middle school in the northeastern U.S. served as the pre-OBPP baseline group, and a second cohort of 112 students in 7th and 8th grades served as the post OBPP group. Intervention began in February, 2007. Baseline assessments were administered in February 2007 to the cohort of 158 students and 17 teachers. The Time 2 assessment occurred in March, 2008 and was administered to the cohort of 112 students and 10 teachers. The baseline cohort included 59 7th graders and 99 8th graders. The post assessment included 49 7th graders and 62 8th graders (these 8th graders included 7th graders from the pre-OBPP baseline cohort).

Sample Characteristics: The pretest cohort included 45.6% female, 38% Black/Multiracial, 7.6% Hispanic, 51% Caucasian, and 3.4% Native American/Asian. The posttest cohort included 44.6% female, 48% Black/Multiracial, 10.5% Hispanic, 35.5% Caucasian, and 5.7% Native American/Asian.

Measures: The Revised Olweus Bully/Victim Questionnaire and the Teacher Questionnaire were used. Reliability and validity of the Olweus Bully/Victim Questionnaire has been previously validated. The Teacher Questionnaire included 29 items that addressed perceptions on prevalence of bullying, locations of bullying, types of bullying, students' reports of bullying, and teachers' and administrations' response to bullying. To be classified as a bully or a victim of bullying, one had to report having bullied or been bullied "2 or 3 times a month" to "several times per week."

Analysis: Pearson chi-square test for independence was used to evaluate statistical significance for items on the Olweus Bully/Victim Questionnaire. A phi coefficient was used to measure the effect size for a chi-square test for independence. Based on the small teacher sample, a Fisher's Exact Test was used to evaluate statistical significance on the teacher interview items.

Outcomes
Posttest: Results were quite mixed on the Olweus Bully/Victim Questionnaire for females. There was a 34.4% decrease in reports of being excluded and 31.1% decrease in reports of being bullied among 7th grade girls. However, contrary to expectations, 8th grade girls were more likely to be physically bullied (20% increase), at a higher frequency (25% increase), and taken part in bullying (35.6% increase). Indirect verbal bullying did decrease among 8th grade girls (35% decrease). There were no significant findings on these variables for 7th or 8th grade males, and the article suggests some negative trends on certain prevalence measures.

Teacher reports indicated significant and positive improvements in their ability to identify, manage, and report bullying incidents.

Study 9

Evaluation Methodology
Design: Through the Pennsylvania Department of Education (PA CARES) and Windber Research Institute (HALT!), the OBPP was provided to 100,000 children in more than 70 school buildings in grades K-12. District-wide implementation became a part of HALT!, and received intensive support and onsite consultations, up to 8 hours per week per public school district as needed (3-7 buildings per district). Districts that chose to implement at the building-level became a part of PA CARES, receiving up to 12 hours of certified trainer support during the first year of implementation.

An age-cohorts design was used to analyze program effects. Data from two equivalent age cohorts of students were compared at two or more points in time. There were a total of 107 schools and 54,128 student responses across both 2007 and 2008. PA CARES conducted their baseline assessment in 2008. (It is assumed that a Time 2 assessment occurred in 2009, but the time period is not clear from the report.) HALT! conducted assessments in 2007 (pre) and 2008 (Time 2) and 2009 (Time 3). The HALT! 2007 schools only included elementary and high schools (no middle schools), although middle schools were included in HALT! 2008 surveys.

Measures: The Olweus Bullying Questionnaire was administered to students in grades 3 to 12 prior to the start of the program and at the end of Year 1 and Year 2. The Teacher Questionnaire was also administered to teachers in HALT! schools.

Analysis: A measure of relative change was used to calculate the difference in percentages between baseline assessment (2007) and the assessment after one and two years of the program (2008 and 2009).

Outcomes
Posttest: Overall, after one to two years of program implementation, and across age groups and cohorts, intervention students reported a decrease in bullying others. However, self-reports of being bullied showed mixed findings with some age groups and cohorts showing decreases and others showing no positive program effects. Positive changes were observed across most age groups and cohorts in students' perceptions that adults in the school were actively working to address bullying. Across most groups, increases were observed in the percentage of students who indicated they would try to help a bullied student, and decreases in the percentage of students who would passively stand by or join in the bullying.

Study 10

Evaluation Methodology

Design: This quasi-experimental study examined 194 elementary, middle, and high schools in Pennsylvania, all of which implemented the program. One cohort of schools completed the baseline assessment and began the program in 2008; another cohort completed the baseline assessment and began the program in 2009. With the program lasting at least one and one-half school years, the posttest assessments followed two years after baseline assessments. The authors refer to Time 1 and Time 3 assessments.

Based on the selection cohort design, the study did not follow students over time but compared the postintervention outcomes for each grade to the preintervention outcomes for the same grade two years earlier. For example, postintervention 6th graders at Time 3 (who were preintervention 4th graders at Time 1) were compared to preintervention 6th graders at Time 1. The design thus compared different cohorts but the same grades.

A total of 70,531 students completed baseline assessments (Time 1), and 63,843 students completed posttest assessments (Time 3). There is no other information on participation rates or the population of students in the schools from which the analysis sample was obtained.

Measures: Subjects completed the anonymous 40-question Olweus Bullying Questionnaire on "students' self-reports of bullying others, being bullied, their own actions when they witness bullying, their attitudes about bullying, and their perceptions of others' actions (e.g., other students and teachers) to address bullying." With most measures dichotomized to indicate bullying 2-3 times a month or more, the questionnaire produced 14 outcomes for analysis. Previous studies showed that the items have high reliability, construct validity, and criterion validity.

Sample Characteristics: At Time 1, the sample included 48.9% girls and 50.5% boys. Students were distributed across grades as follows: grade 3 (11.0%), grade 4 (11.4%), grade 5 (12.2%), grade 6 (15.7%), grade 7 (16.0%), grade 8 (15.9%), grade 9 (5.2%), grade 10 (4.4%), grade 11 (4.1%), and grade 12 (2.3%). The race/ethnicity of the sample was 58.6% white, 4.6% Black/African American, 3.8% Hispanic or Latino, 7.3% other race/ethnicity, 8.6% multi-racial, and 17.2% missing.

Analysis: The analysis combined the schools entering the program in 2008 with the schools entering in 2009. It tested for changes due to the intervention by examining differences over two years between two different cohorts that were in the same grade at preintervention (Time 1) and postintervention (Time 3).

First, the descriptive analysis reported mean probabilities at Time 1 and Time 3 (for dichotomous outcomes. From the probabilities, it calculated the absolute change (Time 1 - Time 3), the relative change (Time 1 - Time 3 / Time 1), and odds ratios (Time 1 odds / Time 3 odds). For continuous outcomes, the analysis used mean values at Time 1 and Time 3.

Second, the inferential analysis used multilevel logistic regression and linear regression to test for the significance of time and the time interactions. The multilevel models adjust for clustering by school.

To supplement the analysis of program effects, an additional analysis checked for history effects in the absence of the program. It compared preintervention outcomes in 2008 to preintervention outcomes of students in the same grade in 2009. If these 2008 and 2009 outcomes differ little, it suggests that changes over time in the absence of the program have limited importance.

Outcomes

Baseline Equivalence and Differential Attrition: With the cohort-based design, the study cannot perform typical tests of baseline equivalence. And with the anonymous questionnaire, the study cannot follow attrition of individual subjects from pretest to posttest. The numbers show 9.5% fewer respondents at Time 3, but comparisons of the gender, race, and ethnic makeup of the Time 1 and Time 3 samples were not provided.

Posttest: All 14 outcomes showed statistically significant improvement from Time 1 to Time 3. With more than 60,000 subjects at each time point, statistical significance provides only a minimal standard. Effects sizes for the improvement over time are all small (though applicable to a large population). The largest odds ratio equals 1.54. Using the formula from Lipsey and Wilson (d = ln(OR)/1.81), this odds ratio is equivalent to .23. The smallest odds ratio is 1.18 (d = .09) and the mean odds ratio is 1.32 (d = .15). The two effect sizes for continuous measures are .09 and .11.

Tests for the interaction of time and student sociodemographic characteristics found many significant results. Despite some differences in program effects, however, nearly all the cohort, gender, race/ethnicity, or grade groups showed significant improvement over time.

As a check on the cohort design, the authors say the following:

To assess possible history effects (i.e., whether observed reductions might be attributed to historical effects and not to the program itself), baseline measurements were compared on key variables at 2008 and 2009. Using Satterhwaite approximation (to correct for unequal sample sizes within schools), the Bonerroni adjustment for multiple comparisons, and a robust estimation (which is recommended when distributional assumptions are not fully met), only one (being socially excluded, adjusted significance = .003) was statistically significant, suggesting that there were no significant improvements in scores on key bullying variables in the absence of the intervention.

Study 11

Evaluation Methodology

Design: The study was conducted in grades 7 to 9 in six Oslo schools in 2001, and the classes were followed up to grade 10. There were four treatment and two control schools. Three cohorts were followed: those starting grade 7 in 1999 were followed from grade 9 through 10; those starting grade 7 in 2000 were followed from grade 8 through 10; and those starting grade 7 in 2001 were followed from then through grade 10. The design thus included two, three or four interviews with the individual subjects, depending on their grade level at baseline. Data collections occurred at the turn of each year. The schools were not chosen randomly, but were stratified on the basis of geographical variation in alcohol drinking among young people in the city. There was no discussion of student turnover. Response rates ranged from 86-90% over the four years.

Sample Characteristics: No information was provided on the sample or differences between the treatment and control schools.

Measures: There were questions on the use and frequency of tobacco, and alcohol over the preceding 12 months (yes/no, number of times), whether the student had been intoxicated in the past 12 months, and use of illegal substances in the past 12 months (cannabis, amphetamine, ecstasy, GHB, LSD, cocaine, heroin).

Analysis: Logistic regression for binary measures and linear regression on aggregated outcomes (proportions in grades) were reported. Multi-level random models were applied for each outcome. In the multi-level models, the measurements for individuals over grades were designated at the lowest level, cohorts at the second level, and schools at the third level.

Outcomes

Posttest: In logistic models, 3 of 9 measures were significant. Control schools had a higher increase in cannabis use and drunkenness (6 times or more and 11 times or more) than OBPP schools. Smoking was marginally significant, with higher increase in control schools. There were no significant differences using aggregated models (fixed and random).

Study 12

Evaluation Methodology
The study was quasi-experimental with data from six secondary schools (three experimental and three control) in the Federal Territory of Kuala Lumpur. In both conditions there was an all boys' school, and all girls' school, and a co-educational school. A bullying questionnaire with 24 items was used to assess physical, verbal, and relational bullying. The Cronbach test for reliability showed acceptable levels.

The report is extremely brief, with no information provided on sample selection or characteristics, baseline equivalency, attrition, or type of analyses. It appears that the analysis involved examining the trend from pretest to posttest within both conditions.

Outcomes
Posttest: Contrary to expectation, there was an upward trend in the three types of bullying (physical, verbal and relational) in the experimental schools as a whole. However, in the all girls' school, there were significant downward trends for the three types of bullying after one year. The trend was upward at the boys' and co-educational schools.

Being a victim of bullying was on a downward trend at the all girls' school on all types of bullying victimization (physical, verbal, relational, signal, and extortion). Victimization was higher at the boys' school, and there were mixed findings at the co-educational school.

Study 13

Evaluation Methodology

Design:

Recruitment: Participants were drawn from schools that were involved in a wide-scale effort to implement the intervention in various elementary, middle, and high schools in 49 counties in southern and central Pennsylvania. Of the 230 schools who implemented the intervention, 210 were included in the evaluation based on having completed the assessment at baseline and two years later. The initial sample included 70,998 students in grades 3-11 from 210 schools. These students represented three different cohorts that began implementing the program in 2008, 2009, and 2010.

Assignment:

The study employed an extended age cohort design, in which posttest results for one cohort are compared to pretest results for the previous cohort when they are in the same grade and same school. For example, to evaluate the effect of the program after one year, students in grade 7 at baseline and grade 8 at posttest were compared with students from the same school who were in grade 6 at baseline and grade 7 at posttest. This design controls for "possible maturational or age-related differences between the comparison groups."

Attrition:

Schools were asked to complete assessments at baseline and annually. In Limber et al. (2018), two sets of analyses were reported: one following 210 schools (n=70,998 students) across two years, and another following a subgroup of 95 schools (n=31,620 students) that completed the assessment across three years. In the two-year sample, of the 70,998 students who completed the baseline assessment, 67,734 completed the Time 2 assessment (i.e., two years after baseline), resulting in an overall attrition rate of 4.6% from baseline to the two-year follow-up. In the three-year subgroup, of the 31,620 students who completed the baseline assessment, 29,814 completed the Time 3 assessment, resulting in an attrition rate of 5.7% for this subgroup.

Sample:

The full sample was 49% female, 50.5% male, and 0.4% unknown. Approximately 12% of the sample was in third grade, 12.1% was in fourth grade, 12.9% was in fifth grade, 16.1% was in sixth grade, 16.9% was in seventh grade, 16.8% was in eighth grade, 4.8% was in ninth grade, 4% was in tenth grade, and 4.3% was in eleventh grade. Sixty two percent identified as White, 5% as Black, 4.5% as Hispanic, 11.5% as Other, and 17% did not report their race. For the subgroup of students whose schools administered the Time 3 (i.e., three years after baseline) assessment, 49.2% were female, 50.4% were male, and .4% did not report their gender. Approximately 14% of the subgroup sample was in third grade, 13.9% was in fourth grade, 14.1% was in fifth grade, 14.2% was in sixth grade, 13.1% was in seventh grade, 13.2% was in eighth grade, 6.3% was in ninth grade, 4.9% was in tenth grade, and 6.1% was in eleventh grade. Sixty one percent of the subgroup sample identified as White, 4.5% as Black, 3.2% as Hispanic, 11.2% as Other, and 20.1% did not report their race.

Measures:

Participants reported on being bullied, bullying others, reactions to bullying, and perceptions of teachers' actions to address bullying using the Olweus Bullying Scale, which was previously self-developed by the researchers and has been used and validated in their prior work. Four outcomes were computed for analysis for the full sample: bullying and victimization (being bullied) when examining the percentage of students who reported being bullied or bullying others two to three times per week or more, and frequency with which students experienced or engaged in nine different types of bullying. Five outcomes were computed for analysis for the subgroup sample: frequency with which students experienced or engaged in nine different types of bullying, empathy for a bullied peer, willingness to join in bullying a disliked peer, and perceptions of class teachers' actions to address bulling. Subscale reliabilities in the current sample ranged from α = .85 (frequency with which students experienced nine different types of bullying two years after baseline) to .89 (frequency with which students engaged in nine different types of bullying three years after baseline).

Analysis:

All program effects were tested separately within grades or clusters of grades (3-5, 6-8, and 9-11, roughly corresponding to elementary, middle, and high school levels). The researchers used multilevel modeling to test for intervention effects while accounting for clustering of students within schools. Program effects were based on school-aggregated outcome variables. Program year (e.g., baseline, one year after program start, etc.) was treated as a within-school treatment indicator. Outcome variables were treated as continuous, with the exception of the two dichotomous questions regarding being bullied/bullying others "2-3 times a month or more (coded 1) versus "less than 2-3 times a month" (coded 0), which were analyzed with multilevel logistic regression using a logit link. The MLR estimator was used in all multilevel analyses except for analyses of possible interaction effects between program and sex with categorical outcome variables. In these cases, the Bayes estimator was used because it is less computationally demanding than MLR. When a significant program x sex interaction effect was found, analyses were rerun on boys and girls separately using the MLR estimator.

Intent-to-Treat: Twelfth grade students were omitted from the analyses "because their numbers were much lower than those in other high school grades (approximately 50% of the numbers of students in grades 9-11)," (pg. 60). In the full sample, complete baseline and posttest data were available for 210 of the initial 230 schools. The remaining 20 schools (n=5980 students, or 7.8% of the total) were dropped. In the subsample of schools with assessments across four consecutive years, complete baseline and follow-up data were available for 95 of the initial 116 schools that had taken the survey up to four consecutive years. The remaining 21 schools (n=8001 students, or 20.2% of the total) were dropped.

Outcomes

Implementation Fidelity:

Successful standard implementation of the intervention was determined by the minimum criterion that a given school had conducted the Olweus Bullying Questionnaire at baseline and two years later, as the surveys "are important components of the program and thereby provide a rough indication that the program had been introduced in the school in a standardized way." In addition, students' perceptions of the degree to which their class teacher actively addressed bullying were assessed. Among all students, there were significant increases in students' perceptions that their primary teacher had increased his or her efforts to address bullying between baseline and Time 1, between baseline and Time 2, and between baseline and Time 3. In addition, among students in grades 3-5, there were significant increases in students' perceptions that their primary teacher had increased his or her efforts to address bullying between Time 1 and Time 3 and between Time 2 and Time 3.

Baseline Equivalence:

No tests of baseline equivalence were reported.

Differential Attrition:

Tests of differential attrition were not reported at the student level. In both the two-year and three-year samples, schools that were dropped for missing data had "somewhat smaller" levels of bullying and being bullied at baseline relative to the remaining schools but were similar on ethnicity and gender. Statistical tests for these differences were not reported.

Posttest:

The program continues indefinitely so no true posttest assessments are available. In the two-year sample, two years after baseline, all grades showed reductions in victimization (when examining the percentage of students who reported being bullied two to three times per month or more), and these changes over time were significant for all but three grades (8^th, 10^th, and 11^thgrades). All but one grade (11^thgrade) showed reductions in the frequency with which students experienced nine different types of bullying. Further, all grades showed significant reductions in the percentage of students who reported bullying others two to three times per month or more. All but one grade (3^rdgrade) showed reductions in the frequency with which students engaged in nine different types of bullying.

Long-Term:

In the three-year subgroup, among students in grades 3-5 and 6-8, there were significant reductions in bullying and being bullied across the three years. These effects were significant at the first posttest assessment and maintained through the second posttest assessment. Among students in grades 9-11, there were significant reductions in bullying and being bullied across the three years. These effects were significant between baseline and Time 3, between Time 1 and Time 3, and between Time 2 and Time 3. For risk and protective factors, among students in grades 3-5 and 6-8, there were increases in expressions of empathy for a bullied peer across the three years, which were significant at all timepoints. Among students in grades 9-11, there were significant increases in expressions of empathy for a bullied peer between baseline and Time 3, Time 1 and Time 3, and Time 2 and Time 3. For students in grades 3-5, 6-8, and 9-11, there were reductions in students' willingness to join in bullying a disliked peer across the three years, which were significant at all timepoints.