Targeted Reading Instruction

Study 1

Summary

Amendum et al. (2011) conducted a cluster randomized controlled trial in eight schools from five school districts. Schools were matched and paired with one school in each pair assigned to the intervention and the other school to a control condition. In all, the study included 43 kindergarten and first grade classrooms. Ten students (5 struggling readers and 5 non-struggling readers) were randomly selected for participation from each classroom. Baseline and posttest assessments measured literacy skills.

Amendum et al. (2011) found at posttest, compared to similar readers in control schools, in intervention schools:

• Struggling readers had higher scores on all four Woodcock Johnson reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds.
• Both struggling and non-struggling readers had higher scores on three of the four reading measures: Word Attack, Word-Letter Identification, and Passage Comprehension.

Evaluation Methodology

Design: The study used a cluster randomized design of eight schools from five school districts in the southwestern United States. Schools were matched and paired before randomization, but one school withdrew after assignment to the experimental group because of problems using the program technology. The seven remaining schools included 26 experimental and 17 control classrooms at kindergarten and first-grade levels. The study offered no information on selection of the schools for participation or the representativeness of rural schools.

Within the classrooms, students with severe disability or unable to speak at least some conversational English were excluded. Of the remaining students, teachers and program consultants used assessments of reading skills and knowledge of the child's progress in school to rank each child from lowest to highest performance. In both the experimental and control classrooms, five children reading below grade level (i.e., struggling readers) were randomly selected for study, as were five children reading above grade level (i.e., non-struggling readers). These two groups together included 10 students per classroom.

The combination of experimental and control schools with struggling and non-struggling readers defined four conditions: 1) struggling readers in experimental schools (n = 112), 2) non-struggling readers in experimental schools (n = 125), 3) struggling readers in control schools (n = 63), and 4) non-struggling readers in control schools (n = 64). Only struggling students in the experimental schools received the treatment, but the professional development of teachers giving the instruction should also help non-struggling readers in the experimental schools.

Baseline assessment occurred in the fall of the school year, and post-intervention assessment occurred in the following spring. Attrition was low. With 364 students having scores at baseline, the sample dropped to 350 (96%) at posttest. The experimental school that withdrew early (but after assignment) because of problems with technology was not included in the assessments and analysis.

Sample Characteristics: All schools received Title I funding. Minority students comprised 57-98% of the students in the schools and most students (57-78%) were eligible for free or reduced-price lunch. The study sample was 36% White/European American, 25% Black/African American, 34% other race (not described), and 52% male.

Measures: The key outcome measures came from four subsets of the Woodcock-Johnson Test of Achievement III battery:

1. Word Attack to measure skills in applying phonetic analysis to pronouncing unfamiliar words,
2. Letter-Word Identification to measure letter and word identification skills,
3. Passage Comprehension to measure understanding, and
4. Spelling of Sounds to measure spelling ability.

Reliabilities for the four scales ranged from .74 to .91.

The Peabody Picture Vocabulary Test - Third Edition, which served as a predictor rather than outcome, measures vocabulary knowledge and has an alpha reliability for elementary students of .92-.95.

Analysis: In an intent-to-treat analysis, multiple imputation was used to deal with missing data. ANCOVAs included sociodemographic variables, the Peabody Picture Vocabulary Test scores, treatment condition, and an interaction between treatment condition and the Peabody Picture Vocabulary Test scores. The models also treated pretest outcome scores as covariates. Planned contrasts were used to make comparisons across subsets of the four conditions.

The authors state that all estimates came from hierarchical linear models that accounted for the nested structure of the data, estimating random intercepts for classrooms and allowing for school-specific variation between classrooms within schools. This procedure dealt with clustering effects both for schools and classrooms. However, it appeared that, outside adjustment of the standard errors for clustering, the randomization variable distinguishing between experimental and control schools was measured at the individual level. Based on a check of the t-distribution, the reported probabilities associated with particular t-values must have used degrees of freedom roughly equal to the number of students (364) rather than the number of schools (7). This may have biased tests in a way that overstated the significance of program effects.

Outcomes

Implementation Fidelity: The teachers participated in four professional-development activities that contributed to program fidelity: 1) a summer institute, 2) weekly or biweekly coaching sessions via the webcam, 3) weekly meetings via webcam about the reading progress of individual children, and 4) monthly or bimonthly webcam meetings to discuss strategies and content for improving the program.

Quantitative measures assessing the duration and quality of program practices came from reports by teachers when meeting with program consultants. The duration measure tapped the total number of weeks spent with an experimental child. The quality measure tapped teachers' use of the tools and faithfulness to program strategies. With the two measures averaged into one, the mean on a scale from 1 to 5 equaled 3.41.

Baseline Equivalence: The study compared all participants (struggling and non-struggling readers) across the experimental and control schools and compared all participants across the experimental and control schools. The comparisons of struggling readers revealed significant differences by race, with white students more likely to attend the experimental schools. The models therefore controlled for race. The comparisons of schools revealed that the experimental schools scored significantly higher at baseline on the Letter-Word Identification scores and Spelling of Sounds scores and marginally significantly higher on Word Attack (p = .07) and Passage Comprehension (p =.13). These differences indicate a pattern of non-equivalence. With only seven schools, deviations from randomness can be expected, and the study controlled for baseline outcomes. Regardless, randomization may not have worked well with the small number of schools.

Differential Attrition: None reported, but the posttest lost only 4% of participants, and missing data were imputed for the small number of students not present for the spring assessment.

Posttest: The struggling readers did better at posttest in the experimental schools than in the control schools on all four reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds. Effect sizes ranged from .40 to .72. For both struggling and non-struggling readers combined, the experimental schools did better than the control schools on three of the four measures: Word Attack, Word-Letter Identification, and Passage Comprehension. Effect sizes ranged from .31 to .61. The effects for all readers suggest that non-struggling readers benefitted indirectly from the program. However, the program worked no better for students scoring low at baseline on the Peabody Picture Vocabulary Test than those scoring high.

Long-term: The study did not gather follow-up data or examine sustained effects.

Study 2

Summary

Vernon-Feagans et al. (2009) conducted a cluster randomized controlled trial in six schools from two counties. Schools were matched and paired with one school in each pair assigned to the intervention and the other school to a control condition. In all, the study included 32 kindergarten and first grade classrooms. Ten students (5 struggling readers and 5 non-struggling readers) were randomly selected for participation from each classroom. Baseline and posttest assessments measured literacy skills.

Vernon-Feagans et al. (2009) found at posttest, compared to similar readers in control schools, in intervention schools:

• Both struggling and non-struggling readers had higher scores on the two reading measures: Word Attack and Word-Letter Identification.

Evaluation Methodology

Design: The study used a cluster randomized control design with randomization at the school level. Six elementary schools in two poor rural counties in the southeastern United States participated in the study. Before randomization, the six schools were divided into three pairs based on matching characteristics of size, percent free and reduced-price lunch, percent minority, and involvement in the Reading First program. Random assignment selected one school out of each pair for the experimental condition and one school out of each pair for the control condition. No information was provided on how the schools were selected for participation or the representativeness of rural schools more generally.

Within each of the schools, the study examined kindergarten and first grade classrooms. For the six schools, there were 14 experimental classrooms, and 18 control classrooms.

In each classroom, teachers used assessment data and their judgment to identify children whose reading skills were below and above grade level. The study then selected five struggling readers and five non-struggling readers in each experimental and control classroom. Struggling readers in the experimental classrooms received the Targeted Reading Intervention, while other groups received reading instruction as usual. The total number of participants was 276.

The program was delivered to struggling readers in one-on-one (or sometimes small group) sessions by the classroom teacher within the regular classroom. Each session lasted about 15 minutes. The number of sessions per week or school years was not discussed. The pretest occurred in the fall and the posttest occurred in the spring of the study year.

All schools and classrooms participated throughout the study. Of the 276 participants, 270 (98%) had baseline scores and 252 (91%) had posttest scores on key measures.

Sample Characteristics: The sampled students were diverse but generally disadvantaged. Approximately 60% of the students were African American and less than 10% of the mothers had a college degree. Most of the students were eligible for free or reduced-price lunch.

Measures: The key outcome measures came from two subsets of the Woodcock-Johnson Test of Achievement III battery: Word Attack to measure skills in applying phonetic analysis to pronouncing unfamiliar words and Letter-Word Identification to measure letter and word identification skills.

Two other subtests from the Comprehensive Test of Phonological Processing served as predictors of the outcomes: phonological awareness and rapid color naming.

Analysis: The analyses used multiple imputation techniques, under the assumption that data were missing at random, to replace missing data. Random intercept regression models accounted for clustering within classrooms and schools, but as described in Study 1, the analysis was still conducted at the individual level. Perhaps most problematic, the analysis does not mention controlling for baseline outcomes.

Outcomes

Implementation Fidelity: Teachers implementing TRI met face-to-face over three days in a summer institute and met in-person biweekly with TRI consultants during the academic year.

Quantitative measures assessing the duration and quality of TRI practices came from reports by teachers when meeting with program consultants. The duration measure tapped the total number of weeks spent with an experimental child. The quality measure tapped the teachers' use of the tools and faithfulness to TRI strategies. The two scales were averaged into one. The average score of 3 (on a scale of 1 to 5) showed mixed success in implementation of the program.

Baseline Equivalence: Not examined.

Differential Attrition: Not examined, although multiple imputation of missing data might counter bias due to attrition.

Posttest: Struggling and non-struggling readers in the experimental schools scored significantly higher than struggling and non-struggling readers in the control schools on the two achievement test measures of Word Attack (d=.41) and Letter-Word Identification (d=.50). For struggling readers, the experimental group scored significantly higher than the control group on Letter-Word Identification (d=.57), but not on Word Attack.

Another test indicated that struggling readers who scored low on the Rapid Color Naming and Phonological Awareness measures gained the most from the intervention on Word Attack but not Letter-Word Identification.

Overall, the evidence across both outcome variables unexpectedly showed more consistent benefits of the intervention for all students than for struggling readers.

Long-term: The study did not gather follow-up data or examine sustained effects.

Study 3

Summary

Vernon-Feagans et al. (2009) conducted a cluster randomized controlled trial in four schools in rural counties of Texas and New Mexico. Schools were matched and paired with one school in each pair assigned to the intervention and the other school to a control condition. In all, the study included 43 kindergarten and first grade classrooms. Ten students (5 struggling readers and 5 non-struggling readers) were randomly selected for participation from each classroom. Baseline and posttest assessments measured literacy skills.

Vernon-Feagans et al. (2009) found at posttest, compared to similar readers in control schools, in intervention schools:

• Both struggling and non-struggling readers had higher scores on all four Woodcock Johnson reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds.

Evaluation Methodology

Design: The study used a cluster randomized control design with randomization done at the school level. Four elementary schools in rural counties in Texas and New Mexico were selected to participate. Before randomization, the four schools were divided into two pairs based on matching characteristics of size, percent free and reduced-price lunch, percent minority, and involvement in the Reading First program. Random assignment selected one school out of each pair for the experimental condition and one school out of each pair for the control condition. No information was provided on how the schools were selected for participation or the representativeness of rural schools more generally. Within each of the schools, the study examined kindergarten and first grade classrooms. For the four schools, there were 26 experimental classrooms and 17 control classrooms.

In each classroom, teachers used assessment data and their judgment to identify children whose reading skills were below and above grade level. The study then selected five struggling readers and five non-struggling readers in each experimental and control classroom. Both struggling and non-struggling readers in the experimental classrooms received the Targeted Reading Intervention, while both groups of readers in the control schools received reading instruction as usual. The total number of participants was 359.

The program was delivered to struggling readers in one-on-one (or sometimes small group) sessions by the classroom teacher within the regular classroom. Each session lasted about 15 minutes. The number of sessions per week or school years is not discussed. The pretest occurred in the fall and the posttest occurred in the spring of the school year.

All schools and classrooms participated throughout the study. The study mentioned missing data but gives no specific figures on students who left the study or failed to complete pretest and posttest measures.

Sample Characteristics: The sampled students were diverse but generally disadvantaged. About one-third of the students were white, 25% African American, and 35% other. Additionally, less than 15% of the mothers had a college degree. Most of the students were eligible for free or reduced-price lunch.

Measures: The key outcome measures came from four subtests of the Woodcock-Johnson Test of Achievement III battery: Word Attack, Letter-Word Identification, Passage Comprehension, and Spelling of Sounds. In addition, Peabody Picture Vocabulary Test (3rd Edition) was used as an outcome measure.

Analysis: The analyses used multiple imputation techniques, under the assumption that data were missing at random, to replace missing data. Random intercept regression models accounted for clustering within classrooms and schools, but as described in Study 1, the analysis was still conducted at the individual level. Perhaps most problematic, the analysis did not appear to control for baseline outcomes.

Outcomes

Implementation Fidelity. As in Study 2, teachers implementing TRI met face-to-face over three days in a summer institute. They also met with program consultants during the academic year but did so via a webcam rather than in person.

Quantitative measures assessing the duration and quality of TRI practices came from reports by teachers when meeting with program consultants. The duration measure tapped the total number of weeks spent with an experimental child. The quality measure tapped the teachers' use of the tools and faithfulness to TRI strategies. The two scales were averaged into one. The average score of 3 (on a scale of 1 to 5) showed mixed success in implementation of the program.

Baseline Equivalence. Not examined.

Differential Attrition. Not examined, although multiple imputation of missing data might counter bias due to attrition.

Posttest: The results showed program benefits for struggling and non-struggling readers on the four outcome variables (Word Attack, Letter-Word Identification, Passage Comprehension, and Spelling of Sounds). For both struggling and non-struggling readers, the experimental group scored significantly higher than the control group on three of four measures: Word Attack (d=.35), Letter-Word Identification (d=.34), and Passage Comprehension (d=.61). For struggling readers, the experimental group scored significantly higher than the control group on all four measures: Word Attack (d=.52), Letter-Word Identification (d=.52), and Passage Comprehension (d=.72), and Spelling of Sounds (d=.40). However, no evidence emerged that the benefits of the intervention were higher for struggling readers with lower scores on the Peabody Picture Vocabulary.

Long-term: The study did not gather follow-up data or examine sustained effects.

Study 4

Summary

Vernon-Feagans et al. (2010) conducted a cluster randomized controlled trial in sixteen elementary schools in five rural counties. Schools were matched and paired with one school in each pair assigned to the intervention and the other school to a control condition; however, one school dropped out prior to randomization. In all, the study included 75 kindergarten and first grade classrooms. Ten students (5 struggling readers and 5 non-struggling readers) were randomly selected for participation from each classroom. Baseline and posttest assessments measured literacy skills.

Vernon-Feagans et al. (2010) found at posttest, compared to similar readers in control schools, in intervention schools:

• Struggling readers had higher scores on all four Woodcock Johnson reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds.

Evaluation Methodology

Design: The study used a cluster randomized control design with randomization at the school level. Sixteen elementary schools in five rural counties participated in the study. Before randomization, the sixteen schools were divided into eight pairs based on matching characteristics of size, percent minority, percent free and reduced-price lunch, and involvement in the Reading First program. All the schools received Title I funding. Prior to randomization, one school withdrew because of difficulties with the technology required to effectively implement the intervention. Within each of the 15 schools (75 classrooms), the study examined kindergarten and first grade classrooms.

Within each kindergarten and first-grade classroom, teachers with the aid of the program consultants used mandated state assessment data and classroom performance during the beginning of the school year to identify children as struggling or non-struggling readers. Five struggling and five non-struggling readers were then randomly selected in each experimental and control classroom. The combination of experimental and control schools with struggling and non-struggling readers defined four conditions: 1) struggling readers in experimental schools (n = 194), 2) non-struggling readers in experimental schools (n = 206), 3) struggling readers in control schools (n = 116), and 4) non-struggling readers in control schools (n = 132). Only struggling readers in the experimental schools received the intervention, but non-struggling readers in the experimental schools may benefit as well from the professional development of teachers implementing the program. The total number of participants was 648.

The program was delivered to struggling readers in one-on-one (or sometimes small group) sessions by the classroom teacher within the regular classroom. Each session lasted about 15 minutes. The number of sessions per week or school year was not discussed. The pretest occurred in the fall and posttest occurred in the spring of the study year.

All but one school participated throughout the study. Of the 648 students, some baseline measures contained missing data (for example, one measure had data on 578 or 89% of the students). The study did not report the sample size at posttest or give figures on attrition.

Sample Characteristics: The sampled students were diverse, but generally disadvantaged. About 50% of the children were from minority backgrounds, and half were boys. Maternal education was generally low as the average was a high school degree with just a few mothers with a college degree.

Measures: The key outcome measures came from four subtests of the Woodcock-Johnson Tests of Achievement III battery: Word Attack, Letter-Word Identification, Passage Comprehension, and Spelling of Sounds. In addition, Peabody Picture Vocabulary Test (3rd Edition) was used as an outcome measure.

Analysis: Multilevel ANCOVA models with imputed missing data treated students as nested within classrooms (the school level added no significant variation and was not used). As described in Study 1, however, the analysis appeared to have been conducted at the individual level. The models controlled for baseline outcome scores and sociodemographic characteristics as covariates.

Outcomes

Implementation Fidelity. Teacher fidelity was assessed through weekly online reporting procedures and ratings of quality done over the course of the year by consultants. Fidelity was at or above 80% for all teachers.

Baseline Equivalence. Not examined.

Differential Attrition. Not examined.

Posttest. The key hypothesis of this study compared the struggling readers in the experimental classrooms with the struggling readers in the control classrooms. Controlling for baseline scores, struggling readers in the experimental schools scored significantly higher than struggling readers in the control schools on four of the five achievement test measures: Word Attack (d=.38), Letter-Word Identification (d=.61), Passage Comprehension (d=.58), and Spelling of Sounds (d=.42). Significant differences were not found for the Peabody Picture Vocabulary Test.

A secondary hypothesis concerned the narrowing of differences between the struggling and non-struggling readers in the experimental classrooms. In support of the hypothesis and the effectiveness of the program, differences between the two groups did not reach significance at posttest for three of the five measures. In contrast, differences between struggling and non-struggling readers in the control schools remained significant for four of five measures.

Long-term: The study did not gather follow-up data or examine sustained effects.

Study 5

Summary

Vernon-Feagans et al. (2018) randomized 100 classrooms in 10 schools. Within each classroom, students found to be at risk of struggling with reading during beginning-of-year screenings were included in the sample (N = 556). Results for measures of literacy skills were compared for the treatment and control groups after one school year of implementation.

Vernon-Feagans et al. (2018) found, as compared to control students, students in the treatment classrooms showed significant improvements in:

• Letter-word identification
• Word attack
• Spelling of sounds
• Passage comprehension

Evaluation Methodology

Design:

Recruitment: The researchers selected 10 low-income schools in three school districts in a rural part of the southeastern United States. All schools received Title I funding and had a majority of students eligible for free or reduced price lunch. The study included all kindergarten and first grade classrooms in these schools. Within those classrooms, they selected students who were classified as struggling readers through age-appropriate screenings including the AimsWeb, Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and Woodcock-Johnson III. They attempted to include about 3 struggling readers per classroom in each of the two years. In other words, the study focused on the subset of students in these classrooms that were most at risk with regards to reading difficulties. Recruited teachers participated for 2 years whenever possible, but their students were followed for just one academic year.

Assignment: The study used cluster randomization at the classroom level. Over the 3 years of the study, 50 kindergarten and 50 first grade classrooms were randomly assigned to either a treatment or control condition. Of the kindergarten classrooms, 26 were assigned to treatment and 24 were assigned to control. Of the first-grade classrooms, 29 were assigned to treatment and 21 were assigned to control.

The assignment resulted in a total of 67 teachers participating in treatment classrooms and 52 teachers participating in the control group. Note that teachers leaving the schools were replaced, resulting in more teachers than classrooms. The total number of struggling readers identified and included in these classrooms was 305 in the treatment condition and 251 in the control group. In the treatment classrooms, teachers received the professional development and implemented the program with the learners identified as needing reading support. Control group participants continued with business as usual.

Attrition: Assessments occurred at baseline in the fall of the school year and at posttest in the spring of the same school year. About 18% of teachers (22% of the treatment teachers and 15% of the control group teachers) left the study between the first and second year of their participation. The authors stated that they left for reasons unrelated to the study. Replacement teachers were trained as needed and maintained in the corresponding condition. Overall student attrition was at 7.2%, or 6.6% of treatment students and 8% of control group students.

Sample: The sample was roughly divided between kindergarten and first grade students and was roughly 55% male. About 54% of the students were Black, 25% were White, and 16% were Hispanic. The authors also provided information on the sample of teachers in the study. Nearly all were female and around 80% were White. The majority had an elementary education certification, and nearly a third had a master's degree. They averaged roughly 9 years of teaching experience.

Measures: To assess students' literacy outcomes, the researchers used four subtests of the Woodcock-Johnson III: letter-word identification, word attack, spelling of sounds, and passage comprehension. The authors cite reliability estimates for these sub-scores from other sources ranging from 0.81 to 0.98 for children from 5 to 7 years of age. Trained graduate students did the assessments, but it was not clear if they were aware of student assignments.

Analysis: Given clustering of students in classrooms and schools, the researchers used 3-level hierarchical linear models (student, classrooms, schools) to estimate program effects. Each of the four outcome measures was examined in a separate analysis with baseline test scores and demographic variables included as covariates. To address concerns over attrition, multiple imputation was used with 20 datasets.

Intent-to-Treat: The researchers used all randomized students in the analyses.

Outcomes

Implementation Fidelity: Research assistants coded a random sample of two videos for each struggling reader in order to assess adherence to and quality of program implementation. About 15% of videos were double coded, resulting in a Cohen's kappa of 0.78 for adherence and 0.84 for quality. Coders indicated that adherence to program activities and strategies was at about 84% and that there was high quality of implementation in 61% of the sessions observed. There was also some evidence that implementation quality improved significantly from the first to the second year of the study. However, dosage was limited. Teachers reported an average of 17 one-on-one sessions with each child in the treatment condition, when the goal had been daily sessions for 6-8 weeks.

Baseline Equivalence: The researchers found no significant demographic differences between the treatment and control condition participants, with tests conducted for both teachers and participating students. They also found no differences in baseline student outcome measures between treatment and control children.

Differential Attrition: Overall student attrition was at 7.2% over the school year of their participation, or 6.6% of the treatment students and 8.0% of the control students. Given the similar attrition rates across conditions, the authors did not present formal tests for differential attrition.

Posttest: After one school year of implementation, as compared to the control group, students in treatment classrooms scored significantly higher on all four outcome measures of letter-word identification, word attack, spelling of sounds, and passage comprehension.

Long-term: Not included