ASSISTments

Study 1

Roschelle et al. (2016) presented main effects, while Murphy et al. (2020) primarily explored how teacher practices and behaviors mediated the impacts of the intervention.

Summary

Roschelle et al. (2016) and Murphy et al. (2020) evaluated the program in a cluster randomized controlled trial. The 46 participating schools with 3,035 students were randomly assigned to either the intervention condition or a matched standard practice/waitlist control condition. Mathematics test scores at the end of the school year served as the outcome.

Roschelle et al. (2016) and Murphy et al. (2020) found at posttest that, compared to the control group, the intervention group had significantly higher:

• Math standardized test scores, especially for initially low-achieving students (Terra Nova testing).

Evaluation Methodology

Design:

Recruitment: Participating schools (n=46) were recruited from throughout Maine to represent the range of school sizes and mathematics achievement levels across the state. Schools were recruited using a consistent set of messages and presentations, which were available on a website and as handouts. All participating schools self-selected into the study, expressing a desire to use the ASSISTments technology either immediately or after serving as a waitlist control. Two cohorts of 7th-grade classes participated, one starting in fall 2014 and one in fall 2015.

A generalizability test presented by Murphy et al. (2020) indicated that the sample generalizes to school districts in Maine but not most other states.

Assignment: Participating schools were first blocked on school type (K-8 or other) and then matched into pairs on prior standardized test achievement. One school in each pair was randomly assigned to treatment or a standard practice/waiting list control group. The control group received the ASSISTments intervention two years later, after all assessments had been completed. Prior to assignment, one school withdrew, and its pair was rematched and randomly assigned. After assignment, one treatment school dropped out of the study, and its pair was dropped as well. Ultimately, 45 schools were assigned and 43 actually participated in the intervention and assessment (22 treatment schools and 21 control schools). The sample included 87 teachers and 3,035 students.

Attrition: With the first school year devoted to teacher professional development and program support, the baseline came at the start of the second year and the posttest came at the end of the second year. Specifically, the assessments occurred in October 2014 and May 2015 for the first cohort and in October 2015 and May 2016 for the second cohort. At the school level, there was an attrition rate of 6.5%. Roschelle et al. (2016) reported that, out of the original 3,035 students in participating schools, 307 either changed schools or skipped the assessment while 122 new students moved into a participating school, resulting in a final analytic sample of 2,850 (94%). Murphy et al. (2020) reported slightly lower figures of 116 new students and a final analytic sample of 2,769 (91%); the difference likely stems from including data from 87 teachers.

Sample:

The student sample was evenly split on gender (49.3% male) but predominantly white (92.6%), with 38.7% of students receiving free or reduced-price lunch, and 12.2% of the student body attending specialized education classes. All students had access to laptops.

Measures:

Student Academic Achievement was measured using different standardized tests at baseline and posttest. The New England Common Assessment Program (NECAP) was used to establish baseline (6th grade) scores in mathematics and reading. At posttest, similar skills were assessed using the TerraNova standardized test, which is widely used across the country as it aligns with the Common Core Standards for Mathematics and reflects national curricula.

Analysis:

The primary analysis in Roschelle et al. (2016) used a two-level hierarchical linear model, with students nested within schools. Some schools used only one mathematics teacher, eliminating the need for nesting students within classrooms, though models were also run with adjustments for the third level of clustering. Analyses also controlled for baseline scores, matching pairs, and available individual and school-level demographic characteristics.

The analysis in Murphy et al. (2020) used a three-level hierarchical linear model, with students nested within classrooms and classrooms nested within schools. Covariates included classroom-level measures such as the mean prior math score for all students, school-level measures such as indicator variables for the matched pairs, and student-level measures such as sixth-grade math score.

Intent-to-Treat:

All students in the target grade were assessed unless they moved away or were out of school on the testing day, though researchers omitted a control group school whose treatment group match dropped out of the study post-assignment.

Outcomes

Implementation Fidelity:

Roschelle et al. (2016) reported that coaches observed each teacher in his or her classroom at least three times during the intervention, but their findings were not reported. In terms of duration, students averaged 14 hours of use-much less than the 24 hours that evaluators anticipated students would use ASSISTments over the course of the semester.

Murphy et al. (2020) reported that the median classroom used the program for 22 weeks, about 55% of the available instructional weeks, which is considered a meaningful duration. Teachers opened 75% of the reports available to them, and students attempted approximately 33 program problems per week.

Baseline Equivalence:

Roschelle et al. (2016) reported no significant differences between conditions on six baseline measures. Murphy et al. (2020) reported two significant differences in eight tests. Treatment students were less likely than control students to be ethnic minorities and to receive free and reduced-price lunch (Table 2, n = 3035).

Differential Attrition:

Roschelle et al. (2016) reported only that the attrition rate in the two conditions differed by 5.6% but did not report whether this difference in attrition rates was significant. There were no tests for differential attrition by student characteristics or baseline-by-condition attrition reported in the article. However, the authors provided further differential attrition results that were requested, which showed no significant baseline-by-condition interactions for four math and reading baseline variables and three demographic characteristics. There was one significant baseline-by-condition interaction for the ethnic minority characteristic.

Murphy et al. (2020) provided several additional details on differential attrition. First, they reported overall student-level attrition of 12.6% and condition differences in attrition of 6.9%. The two figures in combination fall at the boundary between (a) acceptable and (b) acceptable under optimistic assumptions in relation to potential bias as specified by What Works Clearinghouse. Second, joiners - students entering the schools after the start of the program - showed one significant difference in eight tests for baseline equivalence: The treatment group had more ethnic minorities (see Table 3). Third, for the analysis sample that excluded dropouts and included joiners, tests for baseline equivalence of conditions showed one significant difference in eight tests: The treatment group had fewer students qualifying for free and reduced-price lunch - a difference that also emerged in tests of baseline equivalence using the full randomized sample.

Posttest:

Roschelle et al. (2016) found with the two-level model that the intervention group scored significantly higher on the posttest math achievement test than the control group (Hedges' g = .18). After splitting students into low and high-achievement groups, the effect was larger for low-achieving students (g = .29), though both low and high-achieving students had significant gains over control group students.

Murphy et al. (2020) replicated these results in the main effects analysis with the three-level model (Hedge's g = .22). Moderation tests found a statistically significant and negative effect between prior math scores and the treatment condition favoring students entering the study with lower prior math achievement. Mediation tests found no significant indirect effects of the program on math test scores via a measure of targeted homework review practices by teachers.

Long-Term:

The posttest assessment occurred immediately post-intervention.

Study 2

Summary

Koedinger et al. (2010) used a quasi-experimental design to examine 1,240 seventh-grade students from four Massachusetts middle schools. Three schools using the program were non-randomly selected as the intervention group, and one school not using the program was selected as the control group. A state standardized test score at the end of grade seven served as the outcome.

Koedinger et al. (2010) found that, compared to students in the control school and after adjustment for pretest scores, students in the three intervention schools had

• Higher standardized test scores.

Evaluation Methodology

Design:

Recruitment: The sample included seventh-grade students from four middle schools in an urban school district in Massachusetts. The eligible sample of 1,344 students was reduced to 1,240 by examining those 1) with MCAS (Massachusetts Comprehensive Assessment System) scores from both 2006 (6th-grade test) and 2007 (7th-grade test), and 2) whose math teacher assignment could be determined either from ASSISTments use or other test data.

Assignment: This quasi-experiment study non-randomly selected three schools that had adopted the program to serve as the intervention group (n = 947 students) and one school that had not adopted the program because of an inadequate number of computers to serve as the control group (n = 293 students). There was no matching. Intervention students used the ASSISTments program, while control students worked on traditional textbook activities.

Assessments/Attrition: The posttest came at the end of seventh grade, or after one year of the program. The analysis sample with complete outcome data was 92% of the eligible sample.

Sample:

The study did not present figures on race, ethnicity, gender, or free-lunch eligibility, but it noted that regular students made up 79% of the sample and special education students made up 21% of the sample.

Measures:

The single outcome measure came from the score obtained on the Massachusetts Comprehensive Assessment System, a standardized knowledge test. No other information was provided, and it is unclear if the test was limited to math items.

Analysis:

The analysis used analysis of covariance models with pretest scores and student group (regular vs. special education) as covariates. The models did not adjust for clustering within schools, the unit of assignment.

Intent-to-Treat: The analysis used all students with complete data, though it did not impute missing data.

Outcomes

Implementation Fidelity:

Not examined.

Baseline Equivalence:

The study presented no figures for demographic measures, but the authors stated that the two conditions "had similar student characteristics (race, gender, limited English proficiency (LEP), free lunch eligibility, special education students) and teacher/school characteristics (licensed in teacher assignment, percent of core classes taught by 'highly qualified teachers,' student/teacher ratio)." For the outcome, the authors stated, and Table 1 confirmed, that the pretest score was substantially higher for the control group.

Differential Attrition:

Not examined.

Posttest:

The adjusted mean from the analysis of covariance was significantly higher for the intervention group students than the control group students (d = .23). Moderation tests showed a larger effect for special education students than for regular students.

Long-Term:

Not examined.