Whole Number Foundations Level K (Roots)

Study 1

NOTE: Whole Number Foundations Level K was initially known as ROOTS. The original name of ROOTS is used in this study.

Summary

Clarke, Doabler, Smolkowski, Nelson et al. (2016) assessed Whole Number Foundation Level K's efficacy on kindergarten mathematics achievement using a randomized block design of the 290 lowest-achieving students across 29 classrooms. Students at-risk in mathematics within classrooms were randomly assigned to either a very small, 2-student group treatment condition (N=58), the normal 5-student group treatment condition (N=145), or a no-treatment control condition (N=87). Measures were collected at the beginning and end of kindergarten and again in the winter of first grade (6 months after program completion), with 93% of students providing at least some posttest data and 68% retained at 6-month follow-up.

Clarke, Doabler, Smolkowski, Nelson et al. (2016) found that, compared to a similarly low-achieving control group, students receiving Whole Number Foundations Level K exhibited significantly greater improvement at posttest in:

• Discrete number sense skills
• Oral counting
• Early mathematics achievement
• Whole number understanding

Evaluation Methodology

Design

Recruitment: A total of 37 kindergarten classrooms in 14 schools from 4 school districts in Oregon agreed to participate in the study. Schools targeted for recruitment were primarily those that received Title 1 assistance (i.e., federal funding for schools that serve a large percentage of students from low-income families). Of the 37 classrooms, 33 provided a half-day kindergarten program and four provided a full-day program. Twenty-eight certified teachers taught the 37 classrooms - nine of which taught two half-day classrooms (i.e., AM and PM).

All classrooms operated 5 days per week and provided mathematics instruction in English.

Students in these classrooms were screened for eligibility using two standardized assessment of early mathematics, Assessing Student Proficiency in Early Number Sense and the Number Sense Brief. Students scoring less than a 20 on the Number Sense Brief and a composite score placing them in the "strategic" or "intensive" range on the other measure were the targets of the small-group interventions. The scores were then aggregated into an overall composite, and the 10 lowest-achieving, targeted students in each classroom were made eligible for random assignment. In some classrooms with histories of high mobility, the 11^thor 12^th lowest-achieving students also entered the study as alternates. Of the 850 kindergarten students enrolled in participating classes, 290 (34%) met inclusion criteria and consented to enter the evaluation.

Not all classrooms had an adequate number of eligible students, so 14 classes were combined until they yielded an adequate number of students for randomization to treatment or control conditions. This cross-class grouping procedure was applied six times. In four instances using eight total classrooms, sets of two classrooms were combined creating four ROOTS classrooms, and in two instances using six classrooms, sets of three classrooms were combined creating two ROOTS classrooms. Collectively, 29 "ROOTS" classrooms participated (23 + 4 + 2).

Assignment: After screening and blocking on classrooms, the 10-12 lowest performing students from each participating classroom were randomly assigned by classroom to receive the 2-person small-group ROOTS intervention (N=58), the 5-person ROOTS intervention (N=145), or the no-treatment control condition (N=87). The unequal numbers assigned to each group may imply that students were assigned to achieve an approximately equal number of groups rather than students, though this was not explicitly discussed in the study. Control classrooms received core mathematics instruction, which consisted of a variety of published and teacher-developed math programs including Everyday Mathematics, Investigations, and Saxon Math. Observational data revealed that these classes delivered a comparable amount of core mathematics instruction per day (~30 minutes), but in no cases implemented ROOTS or any additional math instruction to control for the additional instruction given to ROOTS students.

Attrition: Students were assessed prior to randomization and at posttest, with one additional measure taken at 6-month follow-up. Most students completed measures at pre and posttest, and while missingness varied across measures, only 6.9% of students (N=20) were missing all posttest data. Attrition was a larger problem at 6-month follow-up, with only 68% of the sample (N=197) retained.

Sample

Participating students were mostly white (58%) females (43%) with an average age of 5.2 years. A third identified as Hispanic (33%), with a similar percentage claiming limited English proficiency, while relatively few were classified as being eligible for special education (11%). Interventionists delivering the program were nearly all female (98%) and white (94%), with the majority (61%) holding a bachelor's degree or higher. They had an average of 8 years of experience and 20% held a current teaching license.

Measures

All measures came from student self-assessments at baseline and posttest, with one measure administered at 6-month follow-up only. Trained staff administered all measures, though it was not clear whether these staff were blind to condition.

Whole number understanding, a measure closely linked to the intervention content, was assessed at pre and posttest using the 32-item evaluator-developed ROOTS Assessment of Early Numeracy Skills (RAEN). The study reports that the assessment's "predictive validity ranges from .68 to .83 with widely used measures of mathematics achievement," but does not otherwise report psychometric properties for the current sample.

Discrete number sense skills were assessed at pre and posttest using the Assessing Student Proficiency in Early Number Sense (ASPENS), a series of 3 curriculum-based measures for screening and monitoring progress in kindergarten mathematics. Moderate test-retest reliability and low predictive validity were reported, but neither appears to be characteristic of the present sample.

Conceptual number skills were tested at pre and posttest using the 33-item Number Sense Brief (NSB). The study reports "a coefficient alpha of .84 at the beginning of first grade" from the instrument's authors, but no measures of validity in the current sample.

Early mathematics achievement was measured at pre and posttest using the Test of Early Mathematics Ability (TEMA)-Third Edition, whose developers report high alternate-from, concurrent validity, and test-retest reliabilities. However, no psychometric properties were reported for the current study.

Problem solving and procedural mathematics ability was assessed at 6-month follow-up, only, using the kindergarten version of the Stanford Achievement Test-Tenth Edition. The widely used standardized achievement test has adequate reported validity and reliability, though neither measure was reported for the present sample.

Analysis

Students in the ROOTS groups were combined to compare their math gains relative to students in the control group. Intervention impacts were assessed using multilevel models with adjustments for repeated (baseline) measures and clustering in intervention groups. Robustness checks included models that further adjusted for clustering at the classroom (a partial unit of randomization, along with the individual) and school level, which returned similar results to those presented. Models were fitted using restricted maximum likelihood estimation, which uses data from all participants providing valid measures at one or more time points; however, the 6-month follow-up assessment had no baseline measure and results were based only on those retained for the assessment. Additional models were used to assess the intervention groups' performance against all typically achieving students (N=560) using similar methods.

Intent to Treat: The study made use of all available data in adherence with intent to treat.

Outcomes

Implementation Fidelity: Several observational measures of fidelity were collected, with research staff finding that interventionists, on average, taught the majority of prescribed activities (rating 4.2 of 5). Dosage was also consistent across the 58 small intervention groups, with all groups receiving ROOTS for 5 sessions per week and 57 of the groups completing 97% or more of the prescribed lessons. The remaining group still completed 88% of activities.

Baseline Equivalence: The study reports "no statistically significant differences at pretest" on pre and posttest outcome measures and an examination of demographic characteristics presented in Table 1 (pg. 612) reveals no substantive differences across conditions. Still, no statistical tests were conducted to examine demographic differences at baseline.

Differential Attrition: There was no assessment of differential attrition for overall pretest scores or demographics, but overall attrition rates did not differ by condition. Analyses of condition-by-baseline attrition revealed significant differential attrition on the Number Sense Brief (NSB) measure of conceptual number skills, with the control group having substantially higher attrition of higher-achieving students than the treatment group. There was also differential attrition on the ROOTS-developed Assessment of Early Numeracy Skills (RAEN) measure of whole number understanding, with those retained in the control group scoring higher than the baseline sample while the opposite was true of the treatment group. The authors note that because of these differences, "results for the NSB total scores and the RAENS should be interpreted with caution."

Posttest/6-Month Follow-Up: Compared to the similarly-achieving control group, those receiving either ROOTS intervention improved measures of discrete number sense skills (g =.58), oral counting (g =.28), early mathematics achievement (g =.32), and whole number sense (g=.75) at posttest. There were no differences between groups on the single item assessed at 6-month follow-up, and no tests were presented comparing the 2-person ROOTS groups to the 5-person ROOTS groups.

Tests comparing those receiving ROOTS against all regularly-achieving students also showed significantly greater improvement in the ROOTS group on the measures of discrete number sense skills (g=.58) and oral counting (g=.28).

Study 2

NOTE: Whole Number Foundations Level K was initially known as ROOTS. The original name of ROOTS is used in this study.

Summary

Clarke, Doabler, Smolkowski, Baker et al. (2016) also used a randomized control trial of kindergarten students nominated by their teachers to examine Whole Number Foundation Level K's efficacy. A total of 29 classrooms were randomly assigned to the program or to a control group, with 140 students nominated by teachers to the treatment (N=67) or control (N=73) conditions. Students were assessed at the beginning and end of kindergarten, with an 11% attrition rate.

Clarke, Doabler, Smolkowski, Baker et al. (2016) found that, compared to a similarly low-achieving control group, students receiving Whole Number Foundations Level K exhibited significantly greater improvement in:

• Early mathematics achievement

Evaluation Methodology

Design

Recruitment: A total of 29 kindergarten classrooms from two school districts in the Pacific Northwest agreed to recruit students for the study. Within these classrooms, teachers were asked to nominate the 5 lowest performing students, "or those who would most benefit from a small-group math intervention," to participate. Teachers nominated a total of 140 students, with most teachers (79%) selecting 5 students, though the number selected in each class ranged from 3 to 6.

Assignment: Classrooms were randomly assigned to treatment (N=67 students in 14 classrooms) or control (N=73 students in 15 classrooms) conditions. Both conditions also received another program, Early Learning in Mathematics (ELM), offered as part of the regular school math curriculum, and treatment students were taken out of their general education classrooms 3 days per week to participate in ROOTS. Essentially, the treatment group received some of the Early Learning in Mathematics curriculum in addition to the ROOTS program, while the control group received the full Early Learning in Mathematics program only. The authors noted, however, that treatment and control classrooms provided the same amount of daily mathematics instruction. The design blocked on teachers' previous experience teaching ELM (that is, researchers randomly assigned teachers with 1 year of ELM experience to condition and then randomly assigned teachers new to teaching ELM to condition) and schools (i.e., those with multiple classrooms were assigned to condition within school). Students were assessed at the beginning (pretest) and end (posttest) of kindergarten.

Attrition: The overall attrition rate from pre to posttest was 11% for the early mathematics measure and 10% for the early numeracy outcome.

Sample

Over half (~54%) of students were White, while nearly a quarter was Hispanic (~24%). The majority (~53%) were male and English language learners (~64%) who were eligible for free or reduced lunch (47%). The average age of participants was approximately 66 months. Instructional assistants delivering the program were all white and over 90% female. Nearly a third had college degrees and over 90% had at least one year of teaching experience.

Measures

Two standardized measures were proctored at baseline and posttest by trained researchers, though it is not specified whether these researchers were blind to condition:

Early mathematics achievement was measured using the Test of Early Mathematics Ability, whose developers report high alternate-form and test-retest reliabilities. However, no psychometric properties were reported for the current study.

Early numeracy development was assessed using the Early Numeracy Curriculum-Based Measurement, a widely validated instrument for assessing number skills in kindergarten students. While several measures of developer-reported validity and reliability were presented, no figures were calculated for the current sample.

Analysis

Intervention impacts were assessed using multilevel models with adjustments for baseline measures and clustering in classrooms (the unit of randomization). Since only one ROOTS small group was formed per class, the adjustment also applied to clustering under instructional assistants that provided the intervention. Models were fit using restricted maximum likelihood estimation, using data from all participants who completed measures at (at least) one time point. Additional models were used to assess the intervention classrooms' performance, including typically achieving students that did not receive the intervention (N=228-236), against the full, typically achieving control classrooms (N=255-260) using similar methods.

Intent to Treat: The study made use of all available data in adherence with intent to treat protocol.

Outcomes

Implementation Fidelity: "Online logs completed by the 14 instructional assistants who delivered the ROOTS intervention revealed that groups generally completed all 50 ROOTS lessons during the year" (pg. 4), and observers rated the implementation of materials very highly, averaging a 2.92 out of 3 across instructors over 3 time points.

Baseline Equivalence: Tests for baseline differences on outcome measures showed no significant differences between conditions. There were no formal tests for demographics, despite an apparent difference between conditions in the distribution of English language learners (72% vs. 55%).

Differential Attrition: Attrition rates were marginally higher for the control group than the treatment group, though the difference did not meet conventional thresholds for significance (p=.08), and there was no differential attrition on the pretest math achievement overall. Baseline-by-condition analyses revealed no significant differences on either measure.

Posttest: The treatment group exhibited a larger increase (g=.38) in early math achievement than the control group at posttest. Changes in early numeracy did not differ significantly between groups.

Additional analyses compared those who received ROOTS to all typically-achieving students in treatment classrooms. For the measure of early math achievement, the gains made by ROOTS students significantly exceeded gains made by those who did not receive the program, though there was no difference on gains in numeracy.