QuickReads

Study 1

Summary

Vadasy and Sanders (2008a) randomly assigned students in grades 2-5 with poor reading skills to dyads and then randomly assigned dyads to experimental and control conditions. About a dozen schools participated in the randomization, producing a sample size of 188. Scores on multiple measures of reading accuracy, fluency, and comprehension were obtained at pretest in the fall of the school year and at posttest in the spring.

Vadasy and Sanders (2008a) found that the treatment group relative to the control group significantly improved

• Reading fluency
• Reading accuracy.

Evaluation Methodology

Design: A total of 162 students (82 treatment and 80 control) from 13 urban, public elementary schools were used in the evaluation of this Quick Reads program. No information was provided on the selection of the 13 schools. After a recruitment and screening process, eligible students were randomly assigned to student-pairs (dyads) within each grade and school. Then student dyads were randomly assigned to the treatment or control condition.

Student Eligibility: Teachers at the schools were asked to identify students who (a) had never been retained, (b) had low rates of reading fluency or comprehension, and (c) would benefit from fluency intervention. Once active parental consent was obtained from these students, they were screened for eligibility using two grade-level passages from the Oral Reading Fluency subtest of the Dynamic Indicators of Basic Early Literacy Skills. Students whose performance was in the 10th to 60th percentile range (average = 34th percentile) were considered eligible for participation and assigned to dyads. After randomly assigning dyads to conditions, the total sample of 188 comprised 96 students (48 dyads) in the treatment condition and 92 students (46 dyads) in the control condition.

Pretests occurred in the beginning of the school year (fall), the intervention lasted from November to March, and the posttests occurred in the spring.

Attrition: By the end of the study, 14 treatment and 12 control students were lost to attrition (14%). For both the experimental and control groups, the other member of the dyad was removed from analysis if one member moved from the school. The sample used for analysis thus consisted of 162 total -- 82 treatment students (41 dyads) and 80 control students (40 dyads).

Intervention. The two students in the same dyad received the intervention together. The intervention included supplemental fluency instruction in the Quick Reads program delivered by trained tutors. The intervention occurred for 30 minutes per day, 4 days per week for 15 weeks, and most though not all students (70%) missed normal reading instruction to attend the program tutoring. For each student session, tutors recorded attendance and the fluency rate and accuracy for each Quick Reads passage. Control students received regular classroom instruction.

Sample: The sample had the following characteristics: 49% male, 56% minority (13% Asian, 23% African American, 19% Hispanic, 2% other). More than 70% of the students were Title 1, more than 15% were in special education, and more than 20% were learning the English language.

Measures: Students were individually assessed on rapid automatized naming (RAN) and four reading subskills that were hypothesized to be affected by the intervention: word reading accuracy, word reading efficiency, fluency, and comprehension. Reliabilities all exceeded .80 and most exceeded .90.

• Rapid Automatized Naming (RAN) was measured at pretest using the Letter Naming subtest of the Rapid Automatized Naming/Rapid Alternating Stimulus tests.
• Word reading accuracy was measured using the Word Identification subtest from the Woodcock Reading Mastery Test.
• Word reading efficiency was measured using the Sight Word subtest from the Test of Word Reading Efficiency.
• Fluency was assessed using passages from the Dynamic Indicators of Basic Early Literacy Skills and the Gray Oral Reading Test.
• Comprehension was assessed at pre-test and post-test with the Gray Oral Reading Scale Comprehension sub-test.

Classroom Literacy Observations: The investigators expected that the program might work better in classrooms that emphasized oral text reading. To measure this activity, researchers asked 52 classroom teachers for permission to observe midyear reading instruction on two separate occasions. Six teachers (12%) declined to participate. An adapted version of the Instructional Content Emphasis - Revised was used to measure classroom literacy instruction, with a specific measure of time spent on oral text reading expected to facilitate success of the treatment.

Analysis: Due to nesting structures present in the research design, a multilevel (hierarchical) modeling approach was adopted for analyzing group differences on pretests and gains. Pretest-posttest gains were measured two ways: the first set of analyses tested the direct effects of the treatment on the outcomes, and the second set tested the unique effects of the treatment controlling for classroom oral text reading, pretest rapid automatized naming, and word reading accuracy. For the second set of models, oral test reading and pretest scores were grand-mean centered for ease of interpretation. In both sets of models, researchers ignored dyad membership, which had been assigned randomly.

Intention-to-treat: It is unclear whether the study complied with the intent to treat principle. It appears that no intent was made to gather data on those students who left the study during the school year.

Outcomes

Baseline equivalence: There were no significant differences between intervention and control groups on grade or sociodemographic characteristics. For the pretest scores on reading skills, significant differences emerged between groups on one measure: the control group was 2.2 points higher than the treatment group on pretest word-reading accuracy.

Implementation fidelity: A variety of methods were used to ensure fidelity of implementation. Tutors participated in a 4-hour training session that included an overview of fluency development in children as well as instruction in the Quick Reads program. Researchers also observed the tutors and collected data on the tutors' adherence to the Quick Reads protocol, instructional behavior, and student progress. Additionally, evaluators created an instructional package that allowed all interventions to be scripted in order to ensure consistency.

Treatment students read an average of 95 passages, attended an average of 50 tutoring sessions and averaged 1.9 passages read per session. However, classroom teachers reported that most treatment students (n = 58, or 70%) missed some portion of the classroom reading instruction.

Differential Attrition. The study did not compare the 14 treatment and 12 control students who dropped out with those that stayed. It did, however, compare the 13 students missing data on classroom observation because of teacher non-participation. The authors reported no significant differences between these groups on any of the non-classroom observation variables in the model.

Post-test. Comparisons of change in means showed significantly better improvement for the treatment group on four of six reading measures: reading accuracy and three measures of reading fluency (but not reading efficiency and comprehension). Cohen's d ranged between .29 and .43 for the significant effects.

Models that included the classroom behavior measures (with 13 fewer cases because of the non-participation of six teachers) revealed much the same. On average, the treatment significantly improved outcomes on five of six measures (effect sizes ranged from .32 to .65). Time spent on oral text reading, the key classroom observation measure, improved reading performance but did not facilitate the treatment effect.

There was some evidence of three- and four-way interaction effects, but the complexity of the results made it hard to draw implications for program success.

Longterm. Not examined.

Study 2

Summary

Vadasy and Sanders (2008b) randomly assigned students in grades 2-5 with poor reading skills to dyads and then randomly assigned dyads to experimental and control conditions. About a dozen schools participated in the randomization, producing a sample size of 119. Scores on multiple measures of reading accuracy, fluency, and comprehension were obtained at pretest in the fall of the school year and at posttest in the spring.

Vadasy and Sanders (2008b) found that the treatment group relative to the control group significantly improved

• Word comprehension
• Passage comprehension
• Curriculum-based vocabulary.

Evaluation Methodology

Design:
Recruitment/Sample size/Attrition:
In the fall of the academic year, 40 fourth- and fifth-grade teachers in 12 public elementary schools in a large northwestern city were asked to refer students who (a) had never been retained, (b) had low rates of reading fluency or comprehension, and (c) would particularly benefit from a fluency-oriented intervention. Once active parent consent was obtained, referred students were screened for eligibility. Students were considered eligible for participation if they demonstrated at-risk performance on the Oral Reading Fluency subtest of the Dynamic Indicators of Basic Early Literacy Skills. After group assignment (see details below), the sample comprised 70 students in both intervention and control group (N=140). By the end of the year, 16 (23%) treatment and 5 (7%) control students were lost due to relocation, schedule conflicts, removal because of behavior problems, or parental request. The larger attrition in the treatment group was due to the study design in which students were assigned to pairs (dyads). If one treatment student left, the student's dyad partner was also removed from the study, because tutoring instructions were designed for pairs. Controls, however, were dyads in name only, and thus the attrition of a control student did not affect another student. The final sample size was N=119 (Treatment: N=54, Control: N=65). In addition, 20 tutors were recruited from the school communities to implement the program.

Study type/Randomization/Intervention:
This study was a randomized control trial. Group assignment was conducted using a two-stage procedure. In the first stage, eligible students were randomly assigned within schools to dyads (pairs of students). In the second stage, dyads were randomly assigned to one of two conditions: treatment (supplemental Quick Reads tutoring) or control (no tutoring; classroom instruction only). Assignment to dyads occurred across classrooms and across grade-levels.

The authors modified the Quick Reads program, which was originally designed as a teacher-led classroom intervention, to fit the framework of tutor-led instructions for pairs of students. Treatment students were pulled out of the classroom to receive supplemental Quick Reads tutoring in dyads for 30 minutes per day, 4 days per week, for 20 weeks (November to June). Students assigned to the control group received regular classroom instruction only.

Assessment:
Students were individually pretested in the fall of the school year and posttested in the spring by trained research assistants unaware of group assignment.

Sample characteristics:
The sample comprised 4th (55%) and 5th graders (45%) of varying socio-economic background. About 46% of the students were male and 54% were female. A large proportion of the children (76%) were characterized as minority with African Americans, forming the largest minority group, contributing around 40% of children to the study population. A quarter of students (27%) were English Language Learners, and 23% were enrolled in special education programs. Finally, 90% of all children were Title 1 students, indicating the low socioeconomic status of the study population.

Measures:
Students were assessed on the following skills, hypothesized to be affected by the intervention:

• Word reading accuracy: measured using the Word Identification subtest from the norm-referenced, standardized Woodcock Reading Mastery Test-Revised (WRMT-R) (Internal consistency=.93-.94).
• Word reading efficiency: measured using the Sight Word subtest from the norm-referenced standardized Test of Word Reading Efficiency-Form B (Internal consistency=.94-.95).
• Vocabulary: assessed using multiple-choice, curriculum-based measures of vocabulary (Internal consistency=.83-.84).
• Word comprehension: assessed using the WRMT-R Word comprehension subtest (Internal consistency=.92).
• Fluency rate: assessed using the Dynamic Indicators of Basic Early Literacy Skills Oral Reading Fluency benchmarks (Internal consistency=.96-.99).
• Passage comprehension: measured using the WRMT-R Passage Comprehension subtest (Internal consistency=.89).

Analysis:
The study used multilevel models that allowed for heterogeneous group variances as well as random variation between dyads, controlling for baseline scores. Pretest measures were group-mean centered. Since the authors did not observe significant differences between schools on any measure, they chose not to include schools as a third level.

Intention-to-treat: It is unclear whether the study complied with the intent to treat principle. It appears that no intent was made to gather data on those students who left the school or the one student who was dropped from the study.

Of those screened, one fifth grader was recommended for an alternative intervention, as the student's reading skills were "far lower than bottom 10th percentile performance" (p. 237). In addition, a few schools had uneven numbers of eligible students, and "three eligible students were not assigned to either condition (through a random selection process) and were subsequently removed from study participation" (p. 237).

Outcomes - Posttest, Long-term

Implementation fidelity:
Tutors participated in two training sessions (prior to intervention and midyear). To ensure that all tutors used the same procedures, the Quick Reads instruction was scripted. Researchers visited tutors weekly to provide coaching and monitor the quality of implementation. Overall, fidelity was high with an average of 90% adherence to the Quick Reads protocol.

Baseline Equivalence:
Chi-square tests indicated no statistically significant difference between study groups on socio-demographic characteristics. No significant differences between groups were observed on any outcome measure at baseline.

Differential attrition:
No test of differential attrition was performed.

Post-test:
Multi-level models revealed significant treatment effects for 3 out of 7 (43%) outcome measures. The model estimates showed that students in the treatment group had a 3-point advantage over controls on the curriculum-based vocabulary measure (p<.01; d =.42); for word comprehension, the treatment group was estimated as having an advantage of 3 standard score points (p<.05; d = .27); and finally, for passage comprehension, treatment students were estimated as having an advantage of 4 standard score points (p<.001; d = .50). For the norm-referenced measures, the results indicate that the treatment group averaged in the 30th percentile at posttest on both word comprehension and passage comprehension, whereas the control group averaged in the 25th and 10th percentiles, respectively.

Effect size:
Small to medium effect sizes (d=.27-.50) were observed for significant results. However, the authors note that the Cohen's d values need to be interpreted with caution since its computation assumes that the variances of the two groups can be pooled, which is unreasonable given the study's research design.

Long-term effects:
Long term effects of Quick Reads were not investigated by the study.

Study 3

Summary

Trainin et al. (2016) randomly assigned 76 classrooms (1,484 students) within grade level to three conditions: QuickReads print-only instruction (classroom n = 26), QuickReads print + technology instruction (classroom n = 27), and a control with standard district fluency instruction (classroom n =23). Scores on multiple measures of reading fluency, comprehension and vocabulary were obtained at pretest in the fall of the school year and at posttest in the spring for an analysis sample of approximately 912 students.

Trainin et al. (2016) found that the treatment group relative to the control group significantly improved

• Oral fluency
• Reading comprehension
• Vocabulary.

Evaluation Methodology

Design:

Recruitment:

In one Midwestern city, the authors recruited nine elementary schools with 76 classrooms in grades 2-5. No other information was given. The classrooms had 1,484 students who participated in the study.

Assignment:

The 76 classrooms were randomly assigned within grade to one of the three conditions: QuickReads print-only instruction (n = 26), QuickReads print + technology instruction (n = 27), and a control group with standard district fluency instruction (classroom n =23). Then, classroom teachers administered the grade-appropriate Comprehension and Vocabulary tests to all their students involved in the study as a pretest. Each classroom teacher then stratified their students into reading ability levels (high, medium, low). The research team randomly selected 12 focus students (four high, four medium, four low) from each classroom for further assessment, which suggests that the number of students in the analysis was 912. However, the study failed to report the Ns for the analysis.

Attrition:

The author(s) administered one measure three times: in October, in February, and at end of March/beginning of April. They administered two measures twice: once before October and again in May. No information about attrition was given.

Sample:

Of the 1,484 students, 1143 (77.0%) had demographic information (Table 1). About 26.7% were classified as minorities, 5.8% as English language learners, 44.9% as receiving free or reduced-priced lunch, and 15.7% as needing special education (Table 1).

Measures:

The authors measured the behavioral impact of the QuickReads program through three reading measures completed by participants at baseline and after the intervention group completed 19-week program. Although the measures are well known and validated, no figures on reliability and validity were presented for this sample.

(1) The Oral Reading Fluency subtest from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) measured accuracy and reading rate by having students read a passage aloud for one minute. Students took the test in fall, winter, and spring. Test-retest reliability is .92 to .97.

(2) The Comprehension and (3) Vocabulary subtests of the Gates-MacGinitie Reading Test measured students' abilities to comprehend different types of text and grade-level vocabulary through asking students to answer literal questions, make inferences, draw conclusions, and determine meanings of words. Students took these tests in fall and spring. Internal consistency for the Gates-MacGinitie is .96.

Analysis:

For the reading fluency measure, the authors used a three-level hierarchical linear growth model (time of measurement, student, classroom) to compare achievement results across conditions. For the comprehension and vocabulary measures, the authors used a two-level hierarchical linear model (student, classroom), controlling for a student's baseline score.

Intent-to-Treat:

It appears that the approximately 912 students selected for assessment were followed, but without information on the analysis N, the use of an intent-to-treat sample is uncertain.

Outcomes

Implementation Fidelity:

The authors analyzed classroom observations with field notes and teachers' weekly logs. The authors performed an analysis of variance (ANOVA) on fidelity of implementation scores for both intervention groups (print-only and print + technology) with no significant differences. Overall fidelity was reported as high.

Baseline Equivalence:

Based on their hierarchical linear modeling (HLM) results, the three conditions did not differ at baseline on the three outcomes. No other tests were reported.

Differential Attrition:

No information was given, but it is possible that attrition was low.

Posttest:

At the end of the 19-week treatment, both QuickReads intervention groups scored significantly higher than the control group on all three reading measures [Oral Reading Fluency [ES (Hedges' g) = .16], Comprehension [ES = .21], and Vocabulary [ES = .22]. Higher grades showed less growth in rate than lower grades for oral reading fluency.

Long-Term:

Long-term effects (> 1 year) were not examined.