Tools of the Mind

Study 1

Summary

Blair and Raver (2014) used a block-randomized control trial to assign 29 schools to either a treatment or a business-as usual control group. The sample included 759 children in 79 kindergarten classrooms. The study conducted a pretest early in the academic year, a posttest in the spring of the academic year, and a 6-month follow-up in the fall of first grade. It evaluated outcomes measuring executive function, stress response, academic performance, and classroom behavior.

Blair and Raver (2014) and Blair et al. (2018) found, as compared to the control group, the treatment group had significantly better scores at posttest or follow-up on measures of:

• Reading
• Applied problems (math)
• Vocabulary measures
• Classroom aggression and conduct problems
• Executive function: Working memory, reaction speed, rapid naming

Evaluation Methodology

Design:

Recruitment: The study contacted 12 school districts that agreed to participate. The 29 schools within the school districts agreed to be randomly assigned and participate fully in the project. The study then attempted to recruit 6 students in each of 79 kindergarten classrooms through flyers sent to children's homes. The first 6 students whose parents consented to participate were selected. Over the two years of study, the total sample size reached 759.

Assignment: The study utilized a block randomized design. Schools were randomized within blocks balancing school district, the percent of students eligible for free or reduced-price lunch, school size, state test scores, and number of kindergarten classrooms. The study noted an exception "for 2 schools in the participating districts in which all of the district's kindergarten classrooms were housed in one facility. For these two schools, random assignment was at the classroom level and the school was treated as two schools for analysis purposes." Kindergarten classrooms assigned to the control condition (N = 37) continued with business as usual, while classrooms assigned to the treatment condition (N = 42) implemented program activities.

Attrition: The study conducted three assessments, the first around October (used as a pretest although the program would have started at the beginning of the school year), the second around March (used as a posttest although the program would not have ended until the end of the school year), and the last in the fall of the following year (approximately 6-month follow-up). Blair and Raver (2014) reported approximately 5% student attrition (N=34) from fall to spring; these dropouts were replaced in the spring with other students who had been participating in the program since the start. The total sample size was N = 723 in the fall and N = 725 in the spring. However, the smaller sample sizes listed in Tables 1 and 2 for many of the measures indicate additional missing data.

Blair et al. (2018) examined teacher-reported data that had more attrition. Of the 759 enrolled students, 667 (88%) had data in the spring and 554 (73%) had data in first grade. Table 2 additionally shows considerable missing data at the baseline assessment in the fall of kindergarten, with sample sizes ranging from 534 to 616.

Sample: Students eligible for free or reduced-price lunch ranged by school from 5-92%, and about 15% of the schools in the sample were considered high-poverty schools. No information was provided on the characteristics of the sample students.

Measures: Blair and Raver (2014) measured academic ability using the Woodcock-Johnson III (WJ-III) subtest of applied problems (math) and letter-word (reading) at all assessment points, the WJ-III reading vocabulary subtest at posttest and 6-month follow-up, and the Expressive One Word Picture Vocabulary Test at pretest (both measures of vocabulary ability). In addition, the study measured general reasoning with the Raven Colored Progressive Matrices.

Measures relating to executive functions included cognitive flexibility and inhibitory control with the Hearts & Flowers test, Flanker with Reverse Flanker test, and the Dimensional Change test in the NIH Toolbox. The study measured attention control with the dot-probe test and neuroendocrine function by taking two measures of cortisol and alpha amylase at different times of day at all assessment points. Finally, the study measured reaction speed by timing responses on a number of the above tests (it is not clear which).

Teachers provided six student measures (Blair et al., 2018). Note that kindergarten teachers delivered the program, making the posttest assessments non-independent, but that ratings by first-grade teachers, after the end of the program, should be independent. For the first two outcomes, the Teacher Social Competence Rating Scale measured aggression/conduct scores and self-regulation scores. Alpha reliabilities for the two scales across the three assessments ranged from .71-.95. Third, the Strengths and Difficulties Questionnaire measured general behavioral problems (alphas of .84-.91). Fourth, the Emotional Regulation Checklist measured emotional regulation (alphas of .92-.93). Fifth, the Student-Teacher Relationship Scale-Short Form measured teacher-student relationship quality (alphas of .87-.88). Sixth, the Social Skills Rating System rates student academic competence relative to other students in the classroom (alphas of .95).

Analysis: Blair and Raver (2014) analyzed the data using multilevel models with children nested within schools. For all analyses, the treatment versus control group status was included as a binary indicator along with the pretreatment measure of the outcome variable and child-level and school-level covariates. Although the study also analyzed change scores, they did not report them. The study found that data were missing at random, so used full information maximum likelihood (FIML) estimation.

Blair et al. (2018) used mixed linear modeling with children nested within schools, the unit of random assignment. The models included the baseline score and several sociodemographic covariates that varied across conditions at baseline. The FIML estimates addressed potential bias associated with missing data and adjusted standard errors to account for the non-independence of children nested within schools.

Intent-to-Treat: All participants were included at each assessment point with the FIML estimates.

Outcomes

Implementation Fidelity: The study did not provide any discussion of implementation fidelity.

Baseline Equivalence: Page 8 of Blair and Raver (2014) states that "we examined treatment and control group differences on all variables assessed in the fall of kindergarten to determine if differences between the groups were present at baseline. No differences were observed." Blueprints calculations using the means, standard deviations, and Ns in fact show that, for 21 baseline outcome measures listed in Tables 1 and 2 for the Fall, there were no significant differences between the control group and intervention group means.

In contrast, Blair et al. (2018) noted several baseline differences in race, parent education, and eligibility for free and reduced-price lunch - all used as controls in the analysis. However, there were no significant differences between conditions on the outcome variables.

Differential Attrition: Attrition was low in Blair and Raver (2014), and the study reported that missing data were not related to treatment or control condition. The baseline equivalence tests, which could not include the missing data, showed no differences, and full information maximum likelihood estimation adjusted for potential bias in the estimates resulting from missing data.

Blair et al. (2018) stated that those missing data at the spring of kindergarten or at the fall of first grade did not differ from the rest of the sample in regards to treatment status, age, parental education, and the percentage of free or reduced-price lunch eligible children.

Posttest: The results are difficult to follow and Figures 1 and 2 show confidence intervals based on 1 standard error unit, although the text says the figures show 95% confidence intervals.

At posttest, one of the four measures of academic ability showed significant improvement for the treatment group, as compared to the control group: applied problems (d = .13), but reading, reasoning, and vocabulary did not. At the 6-month follow-up, the significant treatment effect for math at posttest fell to insignificance, while the insignificant treatment effect on reading and vocabulary emerged as significant (d = .14 and .10, respectively). The treatment effect on the reasoning measure did not reach significance at either posttest or follow-up.

At posttest, three of five measures of executive function were significantly improved for the treatment students, as compared to the control students: working memory (d = .14), reaction time (d = .12), and rapid naming (d = .08). The posttest measures of stress response did not differ significantly across conditions.

When testing for moderation by poverty status of the school, the study found that the treatment had significant benefits in high-poverty schools for reasoning, vocabulary, and attention. Moderation for the alpha amylase measure of stress response showed significantly lower scores for the treatment group in high poverty schools, but the cortisol measure of stress response showed significantly higher scores for the treatment group in high poverty schools.

Blair et al. (2018) first examined six teacher ratings of students at posttest, finding significant effects on all but academic competence. The intervention group scored significantly lower than the control group on aggression and general behavioral problems and significantly higher on self-regulation, emotional regulation, and teacher-child relationship. Effects were small (d = .15-.19). At the six-month follow-up, when the ratings were independent, only the aggression measure remained significant (d = -.25). Tests for moderation at both assessments found no differences in program impact by school poverty status.

Long-Term: The study did not include a long-term follow-up.

Study 2

Summary

Barnett et al. (2008) assigned students (n=274) and teachers (n=18) randomly to treatment and control classrooms. After randomization, only 224 students consented to participate. The study conducted assessments in the fall and spring of the intervention year and evaluated language and pre-literacy skills and nonverbal problem solving, in addition to problem behaviors.

Barnett et al. (2008) found significant treatment effects for treatment students, as compared to control students on:

• Teacher-rated problem behaviors
• Receptive and expressive language test among Spanish-speaking children (marginal)

Evaluation Methodology

Design:

Recruitment: One preschool in a high-poverty, urban school district in New Jersey was selected for participation. All 274 children in the 18 participating classrooms of 3- to 4-year-old children were invited to participate, of those 80% (n=224) gave consent.

Assignment: All classrooms on one floor were used for the treatment condition (n=7) and all classrooms on the next floor were used for the control condition (n=11). Teachers were randomly assigned to classrooms within blocks to prevent confounding by factors such as teacher certification. Students were also randomly placed into classrooms. However, consent for students came after assignment, and lack of consent was higher for the control group (n = 40, 24%) than for the intervention group (n = 10, 9%).

Attrition: Since randomization occurred before consent was sought, many attritors dropped out before the first assessment point. Of the 224 who consented to participate, three families requested that their children be transferred from Tools to the district curriculum and one family requested the reverse; these four children did not participate in the study. One child in each group moved out of the district prior to assessment, four in each group were not pretested, and six children in the intervention group and five children in the control group had moved. One child in each group was not tested due to absences. At posttest there were 205 subjects (92% of those consenting, 74% of those randomized).

Sample: The sample consisted of 54% 4-year-olds and 46% 3-year-olds. There were fewer females (47%) than males (53%). Within the sample, 69% reported their primary home language as Spanish. The sample was largely Hispanic (92.6%), with a small number of Black (2.2%), Asian (3.7%), and multi-racial (1.5%) children.

Measures:

The study used:

• the Peabody Picture Vocabulary Test (PPVT-III), a test of standard English vocabulary development;
• the Woodcock-Johnson Psycho-Educational Battery-Revised (WJ-R) and its Spanish equivalent, a measure of cognitive abilities and achievement;
• the Get Ready to Read (GRTR), a measure of early literacy (only administered to 4-year-olds);
• the Wechsler Preschool Primary Scale of Intelligence (WPPSI) Animal Pegs subtest, a measure of nonverbal problem solving and visual-motor proficiency;
• the Expressive One-Word Picture Vocabulary Test-Revised (EOWPVT-R), a measure of expressive vocabulary; and
• the IDEA Oral Language Proficiency Test (OLPT), a test of receptive and expressive language skills of Spanish-speaking children.

In addition, teachers rated internalizing and externalizing behaviors with the Problem Behaviors Scale of the SSRS.

Finally, the study used the Early Childhood Environmental Rating Scale - Revised (ECERS), a measure of global classroom quality; Supports for Early Literacy Assessment (SELA), a study of the quality of literacy instruction; and the Preschool Classroom Implementation (PCI), a measure of use of scaffolding by teachers.

Analysis: The study conducted two analyses. The first was a regression analysis with posttest as the dependent variable and pretest, the intervention, and the child's primary language as independent variables. The second analysis was a hierarchical linear model with treatment at the classroom level used as a fixed effect to control for clustering.

Intent-to-Treat: Four students were excluded for transferring to the non-assigned condition. Otherwise, the analysis used all students with data.

Outcomes

Implementation Fidelity: Classroom observations showed that the intervention classrooms fully implemented all environmental features except writing, while control classrooms did not fully implement any of the key environmental features. Coding of videotapes showed Tools teachers, but not control group teachers, were found to direct significantly more "group talk" than "individual talk."

Baseline Equivalence: The study reported no significant differences between conditions at pretest for gender, ethnicity, primary home language, or outcome measures or on a parent questionnaire of prior child care attendance, frequency of being read to in the home, and parent-reported knowledge of numbers, letters, and colors.

Differential Attrition: Twenty percent of those randomized did not consent to participation. Analysis of variance between those children whose parents declined to participate and those who agreed showed no significant main effects of attrition or interactions between attrition and the intervention. Among those who did consent to participate, the study reported no significant differences between attritors and completors on demographic measures.

Posttest: The analysis using the hierarchical linear model (the more conservative analysis) reported a significant intervention effect only for the SSRS measure of internalizing and externalizing problem behaviors (p<.05). The OLPT measure of oral language skills of Spanish-speaking children was marginally significant (p<.10). However, the study reported that because the HLM results showed high correlation with the regression results, and the ICCs were less than .005, the HLM model may not be necessary. In the regression results, the SSRS measure remained significant, as were the Peabody Picture Vocabulary Test and OLPT measures (p<.05).

For classroom measures, the study reported a significant intervention effect for all three measures: Early Childhood Environmental Rating Scale (p=.003), Supports for Early Literacy Assessment (p=.001), and Preschool Classroom Implementation (p=.002).

Long-Term: The study did not include a long-term follow-up.

Study 3

Study 3 may be the same sample as Study 2, but it is difficult to tell from the information provided.

Summary

Diamond et al. (2007) randomly assigned 147 preschool students to either the Tools of the Mind intervention or the district's version of Balanced Literacy. Students participated in the intervention for 1-2 years; posttest assessments were completed in May and June of Year 2. All measures addressed executive function.

Diamond et al. (2007) found, compared to control students, students in the treatment group significantly improved on:

• Executive function

Evaluation Methodology

Design:

Recruitment: One preschool in a high-poverty, urban school district was selected for participation. The sample includes 147 children in 18 participating classrooms. No other information on recruitment is provided.

Assignment: The treatment condition included 85 children and the control condition included 62 children. Stratified random assignment of teachers and assistant minimized confounds due to teacher characteristics.

Attrition: The study does not provide information on attrition in the sample.

Sample: The sample consisted of 54% 4-year-olds and 46% 3-year-olds. There were fewer females (47%) than males (53%). Within the sample, 69% reported their primary home language as Spanish. The sample was largely Hispanic (92.6%), with a small number of Black (2.2%), Asian (3.7%), and multi-racial (1.5%) children.

Measures: The study used the Flanker Task and the Dots Task, both of which are measures of executive function. The Flanker Task included the Reverse Flanker Task, switching the area of focus. The Dots task included the study used the Dots-Incongruent task, which required remembering a rule and inhibition, and the Dots-Mixed task, which combined congruent and incongruent tasks.

Analysis: The study conducted multiple regression analyses with age, gender, curriculum, and years in curriculum as independent variables.

Intent-to-Treat: The study does not provide information on those included or excluded from the analytical sample.

Outcomes

Implementation Fidelity: The study does not provide information on implementation fidelity.

Baseline Equivalence: The study did not measure outcomes at baseline and no information on demographics is provided.

Differential Attrition: The study does not provide information on attrition.

Posttest: The study reports significantly better scores for the intervention children, as compared to the control children, in the Dots-Incongruent Task, the Flanker Task, the Reverse Flanker Task, and the Dots-Mixed Task. Almost twice as many Tools as control children achieved greater than 75% correct on the Dots-Mixed Task. On the Reverse Flanker Task, the control children performed near chance (65% correct), but the Tools children averaged 84% correct

Long-Term: The study did not include a long-term follow-up.

Study 4

Summary

Farran and Wilson (2014) randomly assigned 60 schools, within blocks, to either the Tools of the Mind intervention or the district's standard curriculum. The sample included 877 preschool students. The pretest was conducted with children at the beginning of their preschool year, the posttest at the end of the preschool year, and two long-term follow-up assessments were conducted at the end of the first and second years of primary school. The study used direct measures of pre-academic skills and executive function and teacher-observed measures of language, social behavior, and self-regulation.

Farran and Wilson (2014) found no positive treatment effects and possible negative treatment effects on:

• Oral comprehension

Evaluation Methodology

Design:

Recruitment: The study recruited eligible school districts in two southern states. The sample included 60 schools in 4 districts in one state and one in the other. A total of 877 children consented to participate (498 in the treatment condition and 379 in the control condition). The consent rate in Tools classrooms was 88% and in control classrooms was 76%.

Assignment: The study assigned schools to conditions. The four smallest school districts were treated as individual blocks. The 22 schools in the large, urban district were divided into five blocks based on the number of classrooms in each school and the experience of the teachers. Within each of those blocks, half the schools were assigned to the treatment condition and half to the control condition.

Attrition: Of the 877 children who consented to participate in the study, 866 (98.75%) recorded at least one pretest measure. At posttest, 821 (93.61%) children recorded at least one measure, 811 (92.47%) children at one-year follow-up, and 778 (88.71%) at two-year follow-up.

Sample: The sample consisted of 56% males. The average age was 54.35 months. The sample was 40% White, 26% Black, 25% Hispanic, and had small numbers of Asian and mixed-race children.

Measures: The study used seven subtests of the Woodcock Johnson III Tests of Achievement, a direct measure of pre-academic skills. The study created a composite measure of academic achievement using the seven measures: letter-word identification, spelling, oral comprehension, picture vocabulary, academic knowledge, applied problems, and quantitative concepts. The study used direct measures to assess executive function. These measures were the Dimensional Change Card Sort task to measure attention shifting capabilities, the Copy Design task to measure attention and visual-spatial skills, the Corsi block-tapping task to measure working memory, the Peg Tapping and Hand-Toes-Knees-Shoulders task to measure inhibitory control.

In addition, the study used teacher reports of social behavior, self-regulation, and language ability. The Cooper-Farran Behavior Rating Scales were used to measure interpersonal skills and work-related skills and the Adaptive Language Inventory was used to measure children's comprehension and use of language in the classroom. Finally, assessors reported self-regulatory behaviors using the Self-Regulation Assessor Rating scale.

Analysis: The study conducted three-level nested regression models, with students at Level 1, classrooms at Level 2, and district randomization blocks at Level 3. Analysis included pretest, age at pretest assessment, interval between assessments, gender, English-language learner status, and individualized education program status as covariates.

Intent-to-Treat: The study included all cases for which there was data.

Outcomes

Implementation Fidelity: The study reports high levels of implementation fidelity.

Baseline Equivalence: The study reports no significant differences between conditions on outcome measures at baseline, however there were small but significant differences in race/ethnicity and age.

Differential Attrition: The study reports no significant differences in attrition by condition.

Posttest: At posttest, only the oral comprehension subscale showed a significant treatment effect, favoring the control condition. No other measures of pre-academic skills, direct reports of self-regulation, or teacher or assessor ratings of social behavior showed significant treatment effects.

Long-Term: At the one-year follow-up, the study reported significant negative effects on the letter word and quantitative concepts subscales, in addition to the composite achievement score for the treatment group as compared to the control group. At the two-year follow-up, the study reports a negative treatment effect on the spelling subscale for the treatment group as compared to the control group. At the one-year follow-up, only one direct measure of self-regulation showed a significant negative treatment effect and at the two-year follow-up the measures of copy design and the composite score of self-regulation showed negative treatment effects. Finally, the teacher and assessor ratings showed no significant treatment effects at either the one- or two-year follow-ups.

Study 5

Summary

Solomon et al. (2018) randomly assigned 20 childcare centers, each with one participating classroom (256 students), to receive the Tools of the Mind intervention or a play-based control condition that did not emphasize self-regulation. Child executive function and teacher- and parent-reported child behavior were measured at baseline, eight months after baseline, and post-intervention.

Solomon et al. (2018) found no significant treatment effects for any of the primary outcomes.

Evaluation Methodology

Design:

Recruitment: Participants were recruited from a network of childcare sites operated by YMCA Canada, specifically located in areas where local elementary schools were not scheduled to offer the full-day kindergarten (FDK) program. Based on these criteria, 20 childcare sites (one classroom per site) were eligible to participate. Children from these sites were eligible to participate if they were 3 or 4 years of age, had a sufficient grasp of English, and did not have any serious developmental challenges. Of the 260 children assessed for eligibility, 1 did not meet inclusion criteria and 3 declined to participate. Teachers were eligible if they were accredited Early Childhood Education teachers and were not expecting to take a leave of absence during the study period.

Participants were recruited in two consecutive cohorts due to high attrition mid-way through the program. Cohort A included 195 participants recruited at baseline, and Cohort B included 61 additional participants recruited from eight months after baseline, after randomization had already taken place.

Assignment: In Cohort A, childcare sites were randomly assigned to the intervention or control conditions (n=10 sites and n=106 children in the intervention condition; n=10 sites and n=89 children in the control condition). In Cohort B, children who were already attending school at the participating sites but were too young to participate at the time of recruiting transferred into study classrooms and received the curriculum that was already being implemented (n=42 in intervention classrooms, n=19 in control classrooms).

Attrition: Assessments were conducted at baseline (Cohort A only), 8 months after baseline, and post-intervention (15 months after baseline). Among those in Cohort A, attrition from baseline to Time 2 was 40% (n=78 dropouts of the initial n=195 participants). The authors claim the high rate of attrition was due to children leaving preschool altogether to attend the newly initiated full-day kindergarten program in the area. An additional cohort of n=61 participants was therefore recruited at Time 2, yielding an overall sample of n=178. Attrition from Time 2 to Time 3 was 11% in the overall sample (n=19 dropouts of n=178 at Time 2) and 11% among participants in Cohort A (n=13 dropouts of n=117 at Time 2).

Sample:

The sample consisted of preschoolers attending YMCA childcare centers in Ontario, Canada. There were 105 males and 90 females in Cohort A, whereas there were 33 males and 28 females in Cohort B. The mean age of participants in Cohort A in the intervention condition was 45.1 months and in the control condition was 45.9 months. The mean age of participants in Cohort B in the intervention condition was 42.4 months and in the control condition was 43.5 months.

Measures:

Primary outcome measures included executive function (two well-established behavioral tasks completed by children) and child behavior (two surveys completed by parents and teachers). Children also completed measures of vocabulary, literacy, and numeracy at baseline (Cohort A) or 8 months after baseline (Cohort B).

Executive function tasks included the Day-Night Stroop task, scored using conventional methods, and the Head-to-Toes task, scored using conventional methods as well as an additional method to bring the data "more in line" with the other data. This produced three analysis outcomes from these two tasks. Child behavior surveys included the teacher- and parent-reported Strengths and Difficulties Questionnaire, from which the Total Difficulties Score was used for analysis, and the teacher-reported Social Competence and Behavior Evaluation Scale, from which three subscales were used for analysis (anger/aggression, anxiety/withdrawal, and social competence). This yielded eight total outcomes computed for analysis: three child executive function outcomes (based on scores from the Day/Night and Head-To-Toes tasks), parent- and teacher-rated behavioral difficulties, teacher-rated anxiety/withdrawal, teacher-rated anger/aggression, and teacher-rated social competence. No reliability information was reported for the present study sample.

Analysis:

Main analyses were conducted using multilevel modeling (MLM) to account for nesting of the data, in which children (level 1) were nested within site (level 2). Due to the relatively low number of sites, Bayesian estimation was utilized and allowed for the inclusion of participants with missing data as well as non-completers. In examining the main effect of curriculum (i.e., intervention vs. control) on each of the primary outcomes, child sex, child age at final assessment, and initial levels of the outcome being tested were included as covariates. All analyses were conducted on Cohort A only as well as Cohorts A and B combined and did not change any pattern of results.

Intent-to-Treat: All available data were used in all analyses. Bayesian estimation was used to account for missing data.

Outcomes

Implementation Fidelity: Intervention adherence was measured using the Tools Implementation Checklist (TIC) completed by members of the research team at three time points (7, 11, and 14 months after baseline).

Baseline Equivalence: For Cohort A, teacher-rated anxiety/withdrawal was significantly higher among children in the intervention condition at baseline relative to children in the control condition, though the effect size for this difference was small. No significant baseline differences were found between children in the intervention and control conditions for Cohort B.

Differential Attrition: For Cohort A, attrition rates were similar in the intervention group (44%) and the control group (49%), and attrition did not vary by age or any of the baseline outcome measures. For Cohort B, attrition rates were similar in the intervention group (12%) and the control group (5%), though whether attrition was related to age or baseline outcome measures was not examined for this cohort.

Posttest: No significant intervention effects were found for any of the primary outcomes tested. Moderator analyses revealed no evidence that intervention effects varied as a function of children's initial language ability, but effects did vary by initial hyperactivity/inattention, such that children in the intervention condition with high initial hyperactivity/inattention showed significantly greater improvements in one of three executive function measures compared to children in the control condition with high initial hyperactivity/inattention.

Long-Term: Not examined.

Study 6

Summary

Diamond et al. (2019) used a clustered experimental design to examine 352 kindergarten students in 18 kindergarten classrooms. Schools were randomly assigned to either the intervention group or the business-as-usual control group. Classroom teachers completed a baseline questionnaire about each child's attitudes and behaviors in the fall of the kindergarten year and standardized assessments measured academic performance. The posttest occurred in the spring of the kindergarten year. The primary outcomes included measures of child academic performance, behavior, and self-regulatory skills.

Diamond et al. (2019) found, compared to the control group, kindergarten students in the intervention group showed significantly improved:

• Reading
• Writing
• Prosocial behavior
• Attention-regulation
• Self-regulation

Evaluation Methodology

Design:

Recruitment: Participants were kindergarten students from public schools in Vancouver and Surrey, Canada. All schools in these two school districts were invited to participate, and both principals and teachers had to agree to implement the program if selected for the treatment group. Kindergarten classes in 20 schools and with 352 kindergarten students were initially included in the study.

Assignment: Prior to randomization, schools were matched into pairs on kindergarten teacher's years of experience and training, student ethnicity, percent of students eligible for free lunch, and the language used at home. The 20 schools were then randomly assigned within pairs to either the intervention group or control group. The control schools appear to have continued with their usual curriculum of utilizing whole group activities, rewards, and time-outs, which were not used in the intervention group. No classrooms in the control group utilized activities resembling those in the intervention group. However, in order to treat the control and treatment teachers comparably, control teachers were offered 3 days of workshops on topics of their choosing, were provided an assistant for 90 minutes daily, and received an allowance of $1,000 to purchase classroom supplies.

Assessments/Attrition: Assessments occurred at baseline (pre-treatment) in the fall of the kindergarten year, within the first month of school. A posttest came 8 months later, at the end of the program in the spring of the kindergarten year. Two schools that were part of a matched pair dropped out of the study a couple of months into the school year, and were not replaced by other schools, resulting in nine intervention schools (student n = 172) and nine control schools (student n = 180). No other information was provided regarding students lost to follow-up. It appears from the text that incomplete teacher reports on students led to missing data, but the extent of the loss was unclear.

Sample: Across all teachers, the mean years of teaching was 15.5 with a range between 1-29 years. The mean years of teaching kindergarten was 7.5 years with a range between 1-15 years. Across all classrooms, approximately 19.5 students were in each class, 51% of students were female, and the average age at baseline was 5.2 years.

Measures: The primary outcomes of reading and writing skills were measured using British Columbia's objective, standardized assessment tools. However, the study gave no details on the tool or on who did the scoring. Otherwise, several teacher-reported measures of student attitudes and behaviors were obtained: social inclusion and other prosocial behavior, attention-regulation, and self-control. Kindergarten teachers both rated the children on these behavioral and attitudinal measures and delivered the program. Additionally, teachers' feelings about teaching were measured based on a 10-point scale with 1 being excited about teaching and 10 being exhausted and burned out.

Analysis: The effect of the intervention was tested using either a Poisson loglinear model when the outcome was skewed or a binary logistic model when the outcome was dichotomous. GEE models adjusted for nesting of students within schools, and a probability level of .025 was used to adjust for multiple tests. The models controlled for children's English as a Second Language (ESL) status, free-lunch status, and teachers' years of experience. However, only ESL was used in the formal, reported analyses.

Intent-to-Treat: The primary analysis appeared to follow an intent-to-treat approach that used all available data, but the study lacked detail on the original and final sample sizes.

Outcomes

Implementation Fidelity: Not examined.

Baseline Equivalence: The study presented baseline condition means for teacher experience and classroom demographics in Table 1. In 16 tests, two were significantly different. The control group had a larger percentage of ESL students and a smaller percentage of lower-income students compared to the intervention group. For some outcomes, the authors noted that the conditions were similar at baseline but did not present full significance tests for all outcomes.

Differential Attrition: At the school level, one intervention and one control group from the same matched pair were dropped, limiting potential bias related to assignment. At the student level, no formal differential attrition analyses were presented, but it is possible that attrition was low.

Posttest: The analyses showed significant condition differences in 14 of 15 tests which spanned over reading (OR = 0.95-2.67), writing (OR = 2.54-8.70), academic improvement, social inclusion and prosocial behavior (OR = 1.01-2.45), attention-regulation (OR = 2.00-2.14), self-control during unsupervised work (OR = 3.00), and teachers' excitement to return in the Fall (OR = 2.75). There were no differences at the posttest between students in mathematic academic improvement.

Long-Term: Not examined.

Study 7

This study examined only the "play" portion of the intervention and, thus, did not test the full Tools of the Mind program.

Summary

Hsueh et al. (2014) randomly assigned 104 Head Start centers with 461 participating 3-year-old children and 1,299 4-year-old children to either the treatment or control group. The pretest came early in the fall, a posttest in the spring of the next year, and a kindergarten follow-up one year later. The study evaluated social-emotional and behavioral outcomes and pre-academic skills. Although 3-year-olds and 4-year-olds participated in the study in mixed classrooms, they were analyzed separately.

Hsueh et al. (2014), Mattera et al. (2013) and Morris et al. (2014) did not find any significant positive results for child, teacher, or classroom outcomes among 3-year-olds. As compared to control students, four-year-olds had significant intervention effects on:

• Teacher scaffolding
• Literacy in the classroom
• Emotion knowledge (child)

Evaluation Methodology

Design:

Recruitment: Head Start centers that are part of a larger Head Start Cares program volunteered to participate in the program. To select a diverse sample, the study included centers in 10 states and 4 regions of the country. The study separately examined 3- and 4-year-old children in 4-year-old and mixed 3- and 4-year-old classrooms. Among 4-year-olds, 79% of parents consented at baseline to have their children participate, and 90% consented by the posttest.

Assignment: After being blocked on race/ethnic composition and part-day/full-day schedules, centers were randomly assigned (N = 104) to one of four conditions: Tools of the Mind, a control, and two other interventions. Analysis was then conducted with the separate interventions and, for 3-year olds, with a treatment group that combined all three interventions. The control group included 220 3-year-old children and the Tools of the Mind treatment group included 241 3-year-old children. For 4-year-old children, the control group consisted of 621 children and the Tools of the Mind treatment group consisted of 678 children.

Attrition: For the study of 3-year-olds, the child pretest occurred in the fall and the posttest in the spring. The study reported approximately 12% attrition in the Tools treatment group and 19% attrition in the control group among 3-year-olds. However, Table B6 (page 69) lists 211 control children and 213 Tools children at baseline, while the analysis results in Table 8 (page 41) list the full sample of 220 control children and 241 Tools children.

For the study of 4-year-olds, the child pretest in the fall and posttest in the spring were supplemented by a 1-year follow-up in the spring of kindergarten. The design allowed for children who did not have consent at baseline to be added for the posttest and follow-up when parents gave consent late. Figure 3.1 thus shows 2,128 children who consented, were selected, and were in the center at baseline, and it shows 868 who were not consented or selected at baseline but were added later. The total equals 2,996. At posttest, the number remaining in the centers was 2,670 (89%). The study thus reported 11% attrition (p. 249). The study tracked 2,599 (87%) into kindergarten.

Note that the pretest occurred from September to December, possibly months after the program began.

Sample: Children in the 3-year-old sample were an average of 3.47 years old and roughly half female (50.46%). Children in the 4-year-old sample were an average of 4.42 years old and roughly half female (48.82%). No information is provided on race/ethnicity of the 3-year-old child sample. The sample of Head Start centers suggests socioeconomically disadvantaged participants.

Among 4-year-olds, the sample was 43% Hispanic, 33% Black, and 16% White. The sample shows evidence of being disadvantaged; 59% of participating children's families were eligible for the Supplemental Nutrition Assistance Program (SNAP) and the average income was $1,800 per month.

Measures: All measures of child outcomes reported for 3-year-olds came from teacher reports. The study used 1) the Behavior Problems Index (BPI), which measures the frequency, range, and type of children's behavior problems, and had scales for total problems, externalizing, hyperactivity, and internalizing; 2) the Cooper-Farran Behavior Rating Scales for work related skills and interpersonal skills, 3) Social Skills Rating System, which measures children's ability to regulate their own behavior, assert themselves to solve conflicts with peers, and cooperate, 4) Student-Teacher Relationship Scale, which measures the teacher's perception of the quality of his/her relationship with individual children. Finally, the study examined pre-academic skills, using the Academic Rating Scale of early language and literacy, mathematical thinking, and general knowledge skills.

The study of 4-year-olds supplemented preschool teacher reports with assessments done by "independent observers" and by kindergarten teachers who presumably did not know of group assignment. For child behavior outcomes, the study used Head-to-Toes, pencil tap, and the Cooper-Farran Behavioral Rating Scales (CFBS) to measure executive function and used the Behavior Problems Index (BPI) to measure behavior regulation, and learning behaviors. The study used the facial emotions identification task, emotions situation task, challenging situations task, the social skills rating system (SSRS), and the Cooper-Farran Behavioral Rating Scales (CFBS) to measure interpersonal skills. For pre-academic skills, the study used the Woodcock-Johnson III Letter-Word Identification test of pre-literacy and literacy skills and applied problems (math), the expressive one-word picture vocabulary test (EOWPVT) of vocabulary, and the Academic Rating Scale (ARS) for early language and literacy, math, and general knowledge skills.

To measure teacher outcomes among both 3-year-old and 4-year-old samples, the study used the Adapted Teaching Style Rating Scale (Adapted TSRS) to measure classroom management, social-emotional instruction, and scaffolding, in addition to the Classroom Assessment Scoring System (CLASS), which measured emotional support, classroom organization, instructional support, and literacy focus.

Analysis: The two studies analyzed three-level models for children within classroom and classrooms within centers. The models included fixed-effects dummy variables for the blocks in which centers were randomized, random effects for centers, and controls for baseline outcomes.

Intent-to-Treat: All participants with available data were included at each assessment point, but the study did not follow children who left the centers.

Outcomes

Implementation Fidelity: The study did not discuss implementation fidelity for 3-year-olds. For 4-year-olds, the study stated that implementation fidelity for Tools of the Mind did not meet "satisfactory." Mattera et al. (2013) reported that the intervention provided teachers with materials, training, coaching, monitoring and technical assistance. Coaches spent 90 minutes in each classroom observing and then meeting with teachers an average of 3 times per month. Monthly fidelity logs were completed by coaches where teachers received a fidelity rating on a 1 (low) to 5 (high) scale. Teachers on average received a rating of 2.97 from coaches on the fidelity scale. The study also reported that fidelity improved over time.

Baseline Equivalence: Tests for child age, gender, and outcome measures at baseline in Table B6 found one marginal difference (p < .10) for 3-year-olds: the Tools of the Mind treatment group had a smaller share of girls than did the control group. However, the table includes only a subset of the cases. Tests for teacher and classroom characteristics revealed only one significant difference.

For the 4-year-olds, only one significant child difference (p < .05) emerged from about 30 tests (Table B4). Tests for teacher and classroom differences (Tables B2 and B3) showed only a few marginal differences.

For both 3- and 4-year-olds, the occurrence of pretests after the program start may have affected the baseline equivalence tests.

Differential Attrition: The study of 3-year-old children found lower rates of attrition among the Tools of the Mind treatment group than in the control group (p < .10) and reported (page 95) that higher-risk children were more likely to drop out. Specifically, children who left the centers had significantly higher levels of total behavior problems and externalizing and internalizing behavior problems than did those who stayed. An additional check examined whether differential attrition produced condition differences at baseline among those who remained at posttest. To quote page 97 for 3-year-olds, "Among children who did not leave the sample, there were no statistically significant differences ... between the Tools of the Mind-Play group and the control group." The results suggest that attrition did not produce an unbalanced sample.

For 4-year-olds, tests showed that children who left the centers were more likely to be male, be rated as having lower levels of social behavior, learning behavior, and pre-academic skills, and perform poorly on executive function and cognitive tests than those who stayed (Table Q.1.). Table B5 tests for baseline differences among the analytic sample and therefore reflects the influence of attrition. It shows one significant difference (for the same characteristic significant for the full sample in Table B4) and one marginal difference.

Posttest: For 3-year-old children, the study found no significant posttest differences for the Tools of the Mind classrooms as compared to the control group in the 12 teacher-reported social-emotional outcomes or the 3 teacher-reported pre-academic skill outcomes.

Among 4-year-olds, the study reported no significant intervention effects for either observer or teacher-reported pre-academic skills; 4-year-olds did show a significant intervention effect for the facial emotions and emotions situations tasks (p<.05), but no effects for measures of executive function, behavior problems, learning behaviors, problem solving, or social behaviors.

For teacher and classroom measures, the study did not find any statistically significant posttest differences for 3-year-olds (p < .05) on 17 measures of teacher practice between the Tools of the Mind treatment group and the control group. Of 14 teacher-reported measures of classroom climate, teachers in the Tools of the Mind treatment group reported significantly worse scores on behavior management as compared to the control group. Among 4-year-olds, the study found a significant intervention effect for a literacy focus in the classroom (p<.05) and scaffolding behavior among teachers (p<.05).

Long-Term: The study did not include a long-term follow-up for 3-year-olds and no significant effects were found at the one-year follow-up for 4-year-olds on 14 measures of behavior, academics, and grade retention/special education.