Number Rockets

Study 1

Summary

Fuchs et al. (2005) employed a randomized control trial design to examine 139 students at risk for mathematics difficulty in 10 schools. The students were randomly assigned to tutoring or control conditions and tested at posttest on measures of math skills.

Fuchs et al. (2005) found

• Significant improvement in mathematics performance for at-risk (AR) tutored students when compared to AR non-tutored controls in computation, calculation, concepts/applications, and story problems
• Significantly higher improvement in calculation and concepts/applications for AR tutored students than students identified as not at risk for mathematics difficulty
• A 40% reduction in prevalence of math disability by the end of first grade for AR tutored students as compared to AR non-tutored students

Evaluation Methodology

Design: Ten schools in a metropolitan school district in the southeastern US participated in this study. Six of the schools were Title I funded. All first grade teachers in these schools and 667 students for whom parental consent was obtained were included. Teachers completed a questionnaire on whole class math instruction with the following results: teachers spent an average of 233 minutes per week on math instruction; lessons consisted of 17% review, 27% instructing on new content, 25% on guiding practice, and 24% on independent practice. There were also about 10 minutes of daily homework assigned. While primary guidance on math skill instruction came from using the district's curriculum standards, other sources of guidance included observations of student performance, classroom assessments, and the district basal. Whole-class instruction was the main method used, followed by small-group instruction, individual instruction, peer tutoring, and cooperative group work.

Students were identified as at-risk (AR) or not-at-risk (NAR) by testing all participating students in a whole-class format on the Curriculum-Based Measurement (CBM) Computation, Addition Fact Fluency, Subtraction Fact Fluency, and CBM Concept/Applications. Testing identified 308 lowest scoring students for individual testing, with an additional 11 nominated by teachers. Individual testing on these 319 children identified 139 as AR. Those students were then randomly assigned to control or tutoring intervention conditions, with blocking by classroom to ensure comparable distribution of AR students in each condition within classrooms. This created 4 pools of students: 1) 69 AR control students, 2) 70 AR tutored students, 3) 180 students who were tested individually then designated NAR, and 4) 348 NAR students only group-tested. Final group numbers (due to attrition) were, respectively: 63, 64, 145, and 292 (groups 3 & 4 were combined into one NAR group).

For the intervention, trained tutors worked with students in groups of two or three students (11 groups of two students, 16 groups of three students). Tutors attended a 2-day training workshop and then received weekly support throughout implementation. Lessons were taught using a scripted manual to promote fidelity. Tutoring sessions were also audiotaped. Each tutoring session lasted 30 minutes, followed by 10 minutes of student use of software called Math Flash, designed to improve automatic retrieval of math facts. There was a sequence of 17 scripted topics completed over 48 sessions (average number of sessions completed by students was 43.8), following the concrete-representational-abstract model. Each session included worksheet and manipulative activities. Each topic took approximately three sessions to complete, and mastery of each topic was assessed daily. Students also completed a cumulative review worksheet of each topic before moving on to the next topic. Student behavior was monitored during each session. Students were awarded points by the tutors for on-task behavior and were able to trade points for prizes. The Math Flash software was designed to provide students with repeated opportunities to hold associations between problem stems and their answers in working memory so the facts could be committed to long-term memory. The program targeted fact families (addition and subtraction facts with answers to 9) that increased in difficulty with mastery. Students were awarded points that translated to "prizes" in a "treasure box" on screen. While efforts were made to minimize the amount of regular math instruction time missed by participating in the tutoring intervention, students missed an average of 10.56 minutes of regular class math instruction.

Sample Characteristics: School characteristics included: teachers with an average of 17.8 years teaching experience and classrooms with an average of 18.32 students. All but one teacher were female and the majority of teachers (34 out of 41) were White (7 were African American). Student characteristics included: generally equal distribution of gender per group; majority of students were White, followed closely by African American, with small percentages of Hispanic and Other. NAR students performed higher on academic and intelligence measures, as expected, with AR students' scores across conditions being comparable.

Measures: Seven math measures were used; five were administered to intact classes and two were administered individually. The whole-class measures included the Curriculum-Based Measurement (CBM) Computation, Addition Fact Fluency, Subtraction Fact Fluency, First-Grade Concepts/Applications, and Story Problems. Scores indicated the number of correct answers on each measure. The two individual measures included the Woodcock-Johnson (WJ) III Applied Problems and WJ III Computation, which also yielded scores of correct answers. Measures were collected at pre- and post-test. In addition, the CBM Computation was administered weekly to whole classes. Reading skill was measured using the Woodcock Reading Mastery Test-Revised. The battery of cognitive assessments conducted in the individual assessments to determine the at risk students (including a teacher rating of attention) are not presented here but are described in the article, and included tests of intelligence, language ability, nonverbal problem solving, phonological processing, processing speed, concept formation, and working memory. Teachers also completed ratings of attention for each AR student. The cognitive variables examined to determine predictors of early math development included attention, language, nonverbal problem solving, phonological processing, processing speed, executive function, and working memory. Analyses were conducted on five of the seven outcome measures (WJ III Applied Problems and Subtraction Fact Fluency were not included because other measures in related domains provided better distributions).

Analysis: One-way ANOVAs were conducted on the pretest, posttest, and improvement scores of the seven math outcome measures, using condition as the factor. Effect sizes were also calculated. Analyses were not intent-to-treat. Multiple regressions were used to determine cognitive variables associated with early math development.

Outcomes

Implementation Fidelity: All tutoring sessions were audiotaped, and program topics 4 and 16 for each tutor were coded and checked for fidelity using checklists corresponding to the lessons. A second coder rechecked fidelity on a random 25% of the tapes, and agreement between coders was 88%. Program implementation fidelity across tutors was 96% for the first topic and 94% for the second topic checked.

Baseline Equivalence and Differential Attrition: There were baseline differences between the not at risk (NAR) and at risk (AR) groups on outcome measures. These differences were expected. There were no significant differences on outcome measures between the two AR conditions. There were a couple of significant differences between groups on the demographic variables. The NAR group had a higher proportion of White students compared to both AR groups, and the AR groups had a larger percentage of students receiving subsidized lunch, compared to students in the NAR group. There was no mention of analysis conducted to determine differences due to attrition between students who left the study (because they moved to a different school) and those who remained.

Posttest: Significant improvements were seen differentially across four of the seven math outcome measures. The greatest effect of the tutoring intervention was seen on the Woodcock Johnson III Calculation and First-Grade Concept/Application measures, where the improvements in AR intervention students exceeded even the students who were not at risk. On story problems, intervention students scored significantly better than the AR control students, although both AR groups were outscored by the NAR students. On the CBM Computation measure, AR students who participated in the intervention saw significant improvements in their scores compared to the AR control students, and their scores were comparable to students in the NAR condition. The status of the conditions on the remaining three measures did not change, with the NAR students performing higher than both AR groups, which scored comparably to each other.

Additionally,based on scores for the Woodcock-Johnson Calculations Test,the researchers identified a 40% reduction in prevalence of math disability at the end of first grade as a result of the math tutoring program (1.77% among non-tutored students vs. 1.06% among tutored students). Among a set of cognitive variables thought to be associated with early math development, the strongest predictor was attention, or distractibility, as rated by teachers. In terms of basic fact fluency, specifically, the unique predictors were attention and phonological processing. For end-of-year computation skill (WJ III Calculation and CBM Computation), attention was the strongest predictor for both, while working memory also predicted CBM Computation. Cognitive variables that predicted math measures in which students worked conceptually with numbers included attention, working memory, and nonverbal problem solving.

Study 2

Summary

Fuchs et al. (2013) employed a randomized control trial design to examine 973 students at risk for mathematics difficulty in 40 schools. The students were randomly assigned to tutoring or control conditions and tested at posttest on measures of math skills.

Fuchs et al. (2013) found

• Significant improvement in mathematics performance for at-risk (AR) tutored students when compared to AR non-tutored controls in simple arithmetic, complex calculations, number knowledge, and word problems
• A narrowing of the achievement gap in simple arithmetic skills between AR tutored students in the speeded practice condition and low-risk controls
• Improvement in use of retrieval as a strategy by AR tutored students, relative to AR non-tutored controls
• Reduction in counting errors for AR students in the speeded practice condition, relative to AR non-tutored controls

Evaluation Methodology

Design: Participants were recruited from a metropolitan school district in the Southeast across four cohorts in four consecutive school years. Forty schools with 233 classes of first graders were included, and students were screened to identify subsets of at-risk students. Of the 4,141 eligible students, 2,806 were screened. Students were excluded if they did not speak English (n=359) or if their standard intelligence scores fell below 80 on (n=59). From the remaining pool, 973 students were randomly sampled (648 at-risk and 325 low-risk), stratified by risk status and classroom. The at-risk students were randomly assigned at the individual level while stratified by class to the three conditions. During the year, there was a loss of 57 at-risk students (8.9%) and 25 low-risk students (7.7%) who moved to schools outside of the district.

Participants were randomly assigned to one of three conditions: a speeded practice condition (n=195), a non-speeded practice condition (n=190), and a control condition (206). A second control group consisting of a sample of low-risk, non-tutored classmates, was also included (n=300). The program was referred to as Galaxy Math, which is related to Number Rockets, and used a space theme. In the non-speeded practice condition, students were encouraged to use a variety of number principle strategies including, but not limited to, relying on number lists, arithmetic principles (cardinality, commutativity principle, and subtraction as the inverse of addition), and efficient counting procedures. In the speeded condition, Flash cards were used and efficient counting strategies were encouraged. In this condition, there was also an emphasis on executing the counting strategies quickly, whereas in the non-speeded condition, the focus was on executing the strategies thoughtfully to emphasize number knowledge. For both intervention groups, tutoring occurred for 16 weeks, 3 times per week, in 30-minute sessions. In both conditions, 25 minutes of each session consisted of number knowledge instruction, with the last 5 minutes consisting of practice, but differing by condition as described above. Tutors monitored behavior and rewarded students for displaying on-task behavior with stickers and prizes.

There were 79 tutors who worked with between four and seven students. Tutors attended a 2-day training workshop and then received weekly support throughout implementation. Lessons were taught using a scripted manual to promote fidelity. Tutoring sessions were also audiotaped.

Sample Characteristics: The students across the three at-risk conditions were predominantly African American (67%-73%) and most participated in the subsidized lunch program (80%-87%). Twelve to seventeen percent were identified as having a disability. Scores across the at-risk sample of students on the screening measures (Wide Range Achievement Test-3 (WRAT) and Weschsler Abbreviated Intelligence Scale (WASI-IQ)) were comparable. The students in the low-risk condition differed in ethnicity, where there were more White students then African American, and a lower proportion of students were on the subsidized lunch program (59%). Only 2% of low-risk students had an identified disability, and on screening scores, low-risk children exceeded those in each of the at-risk conditions.

Measures: Measures were collected pre-intervention and post-intervention. On measures of domain-general cognitive resources, the Woodcock Diagnostic Reading Battery-Listening Comprehension was used to measure the respondent's ability to understand sentences or passages; the Wechsler Abbreviated Intelligence Scale Matrix Reasoning scale was used to measure nonverbal reasoning with pattern completion, classification, analogy, and serial reasoning tasks; the Working Memory Test Battery for Children was used to assess the central executive (listening recall, counting recall, and backward digit recall), the phonological loop (digit recall, word list recall, and nonword list recall), and the visuospatial sketchpad (block recall, mazes memory); the Woodcock-Johnson Reading Battery-III Visual Matching was used to measure processing speed; and the Strength and Weaknesses of ADHD-Symptoms and Normal-Behavior Scale was used to measure attentive behavior. Reliability on all measures was .80 or higher. On the measure of strategic arithmetic behavior, the Addition Strategy Assessment was used to measure response times on simple addition problems, as well as classify the method by which the student solved the problem: counting fingers, verbal counting, retrieval, and decomposition. On measures of mathematics performance, the First-Grade Mathematics Assessment Battery, Arithmetic Combinations was used to measure simple arithmetic problems, while the Double-Digit Addition and Subtraction subsets were used to measure more complex calculations; the Number Sets Test was used to measure number knowledge (understanding of number, numeral mapping, and partitioning sets); Word Problems was used to measure responses using simple arithmetic on word problems. Reliability on these measures was .86 or higher.

Analysis: Analysis was intent-to-treat, but the students from the original sample who lacked data because they moved to schools outside of the school district were dropped from the final sample. A two-level residualized change approach was used to analyze effects of study condition: students were nested in classrooms, and pretest performance was used as a covariate along with the main effect of study condition to predict posttest performance. Effect sizes were also calculated.

Outcomes

Implementation Fidelity: Research staff listened to a randomly selected lesson for each tutor in each condition on a weekly basis, and provided corrective feedback as needed. Additionally, about 16% of the tapes were sampled and coded for fidelity, with a 96% coder agreement. The mean percentage of points addressed was 97.6 in speeded practice and 97.7 in nonspeeded practice.

Baseline Equivalence and Differential Attrition: MCAR testing indicated that missing data occurred at random, so students who left the district prior to the end of the study were dropped without introducing significant bias. For the remaining data, there were no missing data. As noted above, students in the low-risk control condition were different than the at-risk children with regard to ethnicity and participation in the subsidized lunch program, and they exceeded the at-risk children on screening scores (reading and math ability, and intelligence), but there were no significant differences among the at-risk study groups.

Posttest: There were significant improvements among the treatment groups across all four mathematics outcomes. Both tutoring conditions exceeded students in the at-risk control condition on simple arithmetic. Students in the speeded tutoring group outperformed the students in the nonspeeded tutoring group, and they also narrowed the achievement gap with respect to low-risk classmates. Similar results were seen with complex calculations; the only difference being the speeded group did not narrow the achievement gap with the low-risk group. However, the achievement gap widened between the low-risk group and the nonspeeded tutoring and the at-risk control groups. Children in both tutoring groups exceeded the at-risk control group on gains in number knowledge and word problems, and results between the tutoring groups were comparable. However, on these two measures, the students across all three at-risk conditions lost ground with respect to the low-risk students.

On measures of children's strategic behavior when solving simple arithmetic problems, there were significant program effects on counting errors and retrieval. For counting errors, the speeded condition saw greater improvements over the at-risk no-tutoring group than the nonspeeded condition, although gains in both tutoring groups were comparable. However, these gains did not outpace the low-risk control group's gains. Gains between the two tutoring conditions with respect to retrieval were comparable, although only the gains among the speeded practice group kept pace with those of the low-risk control group, whereas the other two conditions lost ground.

Mediation analysis showed an effect of retrieval in mediating the effects of tutoring on arithmetic for both the nonspeeded and speeded practice groups. Evaluation of pretest cognitive resources in predicting improvements in simple arithmetic showed an interaction between nonverbal reasoning and condition. Variables shown to be predictors of posttest improvements were attentive behavior and central executive working memory. Furthermore, these predictors were more significant among the students in the nonspeeded practice condition.

Study 3

Summary

Gersten et al. (2015) and Rolfhus et al. (2012) conducted a cluster randomized controlled trial that examined 76 schools and 994 first-grade students at risk for math difficulties. The study randomized the schools to an intervention group or a business-as-usual control group. A posttest assessment measured several indicators of math skills.

Gersten et al. (2015) and Rolfhus et al. (2012) found that

• Students receiving Number Rockets tutoring scored significantly higher (4.28 points) than control group students in mathematics proficiency as assessed by the Test of Early Mathematics Ability - Third Edition (TEMA-3)

Evaluation Methodology

Design: Participants were first-grade students at risk for math difficulties from 76 schools in four urban school districts across four of the five Regional Educational Laboratory Southwest states, using school as the unit of assignment. Schools were matched within each district on a composite score calculated from mean school achievement scores and the percentage of students receiving free or reduced price lunch. From each pair, one school was assigned to the intervention condition while the other was assigned to the control condition. Eligible students with parental consent were screened on six measures to determine at-risk status: solving computation problems, concept/application problems, brief story problems, number sense, comparative judgments of numerical magnitude, and working memory. Students were deemed at-risk if their composite score was below the sample's 35th percentile. A total of 994 students (615 intervention, 379 control) met this criteria and participated in the study. Intervention students were assigned to tutoring groups of two to three students and tutoring groups met three or more times per week for approximately 17 weeks for a total of 45 lessons. Program implementation ran from December 2008 to May 2009. All students received regular core mathematics. Eighty-six tutors provided program instruction. All were recruited locally and were primarily retired or substitute teachers and had at least a bachelor's degree. Tutors received a one-day training followed by two 2-hour follow-up trainings. Trained district coaches were also available throughout implementation for supervision and technical assistance. Each tutor had an average of 2.8 tutoring groups.

Sample Characteristics: Study participants were predominantly minority (44% Black, 46% Hispanic). About half (49%) of the students were female and 35% qualified for free/reduced-price lunch. About 8% of students had an IEP.

Measures: The Test of Early Mathematics Ability-Third Edition (TEMA-3) was used to measure student proficiency in math, the primary outcome measure. Math skills tested include numbering, number comparisons, concepts, numeral literacy, number facts, and calculation. Reliability of this measure is alpha = .95. The Woodcock Johnson-III Letter/Word subtest was used to measure reading fluency. Measures were collected at post-intervention. Reliability of this measure is alpha = .98.

Analysis: Analysis was intent-to-treat using multiple imputation. Hierarchical linear modeling was used to determine outcome differences.

Outcomes

Implementation Fidelity: Fidelity of implementation was measured in three ways. First, lesson fidelity checklists were used, where tutors audiotaped each session and four sessions were randomly evaluated by coding the checklists. Across all districts throughout implementation, average lesson fidelity was 85%. Next, instructional logs were completed, in which tutors reported on lessons completed and length of lesson. Across study districts, an average of 48.1 lessons was completed, with 32.4% of groups completing all 17 program topics. Finally, tutors collected classroom instruction checklists, where teachers recorded the instructional activities that intervention students would miss. This ensured that intervention students did not miss regular math instructional time, as prescribed in implementation guidelines. Checklists indicated that 11.4% of the classroom activities missed by students for program implementation were the whole-class math instruction.

Baseline Equivalence and Differential Attrition: There was a significant difference in the percentage of consent forms returned from the intervention schools compared to the control schools. There were significant differences between the intervention and control conditions with regard to race/ethnicity and for grade 1 enrollment, which was higher in intervention schools. For students who were identified as at-risk, there were no statistically significant differences found between the intervention and control groups on either demographic variables or mean screener composite scores. Posttest data was available for 90% (n=555) of the intervention group and 86% (n=326) of the control group and there were no significant baseline differences between groups for whom posttest data were available.

Posttest: There was a statistically significant difference found on the main outcome variable, TEMA-3 math proficiency scores, in which students in the intervention group scored an average of 4.28 points higher (ES=.34) than their counterparts in the control group. Exploratory analyses findings indicate that 1) there were no differential impacts of the intervention on students based on baseline math proficiency; 2) there were no differential impacts of the intervention on reading proficiency (as measure by the Woodcock Johnson-III Letter/Word subtest) due to students missing classroom reading instruction for participating in the intervention; and 3) there were no differential impacts of the intervention based on dosage, where student test scores on math proficiency were not significantly higher for students who received more sessions of Number Rockets than students who received fewer sessions of the program.