Literacy First

Study 1

Summary

Caverly and Stoker (2021) used a multisite randomized controlled trial across 22 elementary schools in Texas to estimate the impact of Literacy First on Grades K-2 students' literacy outcomes and longer-term Grade 3 reading and math outcomes for a subsample of Grade 2 students. A total of 520 kindergarten students, 504 Grade 1 students, and 798 Grade 2 students were randomly assigned within schools and grades to intervention or business-as-usual control conditions. Assessments occurred at the beginning and end of school year for Grades K-2 and at the end of the school year in Grade 3.

Caverly and Stoker (2021) found that the intervention students in Grades K-2, relative to control students, had significantly higher:

• Letter sound fluency
• Decoding fluency
• Nonsense word fluency
• Oral reading fluency, and
• Whole word reading

Additionally, in the underpowered sample of Grade 2 students followed to Grade 3, relative to controls, intervention students scored higher (approaching significance) on:

• Reading and mathematics

Evaluation Methodology

Design:

Recruitment: A total of 22 elementary schools in the Austin Independent School District in Texas participated in the study. All Grades K-2 students who scored at the "Tier 2 level" on the beginning-of-the-year assessment (i.e., below grade level) were eligible to receive program services, which included 520 kindergarten students and 504 Grade 1 students. Initially, 1,046 Grade 2 students were eligible across two cohorts, but due to exceeding school tutoring capacities, a total of 798 Grade 2 students were randomly selected for the study and had parental consent.

The study was conducted with one cohort of kindergarten and Grade 1 students and two cohorts of Grade 2 students (in 2018 and 2019). Grade 2 students in Cohort 1 were followed through Grade 3. The original study design followed Cohort 2 through Grade 3, but due to COVID-19, students did not complete the State of Texas Assessments of Academic Readiness (STAAR) in the 2019-20 school year. Therefore, Grade 3 students were Cohort 1 of the initial Grade 2 sample.

Assignment: In a multisite randomized trial, students in grades K-2 were randomly assigned within grade at each school, to either a treatment group that received Literacy First tutoring (n=260 in kindergarten; n=252 in Grade 1; n=397 in Grade 2) or a business-as-usual control group (n=260 in kindergarten, n=252 in Grade 1; n=401 in Grade 2). Students in the control condition were eligible for other types of assistance typically available to students in the participating schools (e.g., literacy coach support).

Assessments/Attrition: Pretest and posttest assessments were completed at the beginning and end of the school year. Attrition was presented separately for each of the three grades. In kindergarten and Grade 1 samples, less than 1% of the randomized student samples (n=4-5 students) attrited between random assignment and pretest. Overall attrition rates between randomization and posttest were about 16% (n=81) for kindergarten students, 14% (n=71) for Grade 1 students, 21% (n=169) for Grade 2 students (across cohorts 1 and 2), and 39% (n=205) for Grade 2 students followed into Grade 3 (i.e., cohort 1 of initial Grade 2 sample).

Sample:

Overall, the kindergarten students were primarily Hispanic (85%) and economically disadvantaged (94.5%), had limited English proficiency (57.5%), and did not participate in special education (89.5%). Grade 1 students were primarily Hispanic (84%) and economically disadvantaged (92%), had limited English proficiency (52%), and did not participate in special education (88.5%). Most of the Grade 2 students who participated in the study were also Hispanic (88%), and over half the students were female (54%). About 69% of the students in the sample were identified as limited English proficient, and about 10% of the students were receiving special education services. Almost all students in the study were identified as economically disadvantaged (93%).

Measures:

The study used existing administrative data from the Austin Independent School District and primary data collected from students. Administrative data from the school district included STAAR Grade 3 Reading and Mathematics data. Grades K-1 early literacy skills data and Grade 2 oral reading fluency data were collected by the intervention program staff. Grade 2 reading comprehension data were collected by a partner evaluator (American Institutes of Research) staff. Students completed either the English or Spanish versions of the assessments depending on their English proficiency level.

Specific shorter-term outcomes in each grade were letter sound fluency and decoding fluency in kindergarten; nonsense word fluency, oral reading fluency, and whole words reading in Grade 1; and oral reading fluency and reading comprehension in Grade 2. Longer-term reading and math outcomes were assessed a year later in Grade 3.

Analysis:

Two-level hierarchical linear models were used for analyses, with students at level 1 and schools at level 2. Level-1 covariates were student-level background characteristics (race/ethnicity, gender, special education status, and economic disadvantage), students' beginning-of-the-year assessment scores, indicators for prior Literacy First tutoring (Grades 1 and 2 only), an indicator for students who completed the assessments in Spanish, and an indicator for students in cohort 2 (Grade 2 students only). To measure assessment scores on the same scales, students' test scores were converted to standardized z-scores using sample-based means and standard deviations on each of the assessments, separately by English and Spanish version language. Analyses were conducted separately for each grade, K-2, and for the longer-term reading and math outcomes in Grade 3.

Intent-to-Treat: The authors stated that students who took the pretest and posttest assessments in different languages did not have comparable scores and were not included in the analyses. The numbers were small in grades K-1 (less than 1%) and grade 2 (4%) and came similarly from both conditions. However, the long-term outcome analyses through grade 3 dropped 15% because of tests taken in different languages (13% from the treatment, and 16% from the control).

Outcomes

Implementation Fidelity:

Not discussed.

Baseline Equivalence:

Tests for student baseline equivalence used the analysis samples for each of the four grades K-3 (Exhibits/Tables 6-10). For kindergarten, Grade 2 and Grade 3 students, there were no statistically significant differences between students in the treatment and control groups on any of the student characteristics or outcomes. For Grade 1 students, one of 17 tests showed differences at baseline, with more White students in the treatment compared to control group.

Differential Attrition:

The authors reported differential attrition rates between study conditions for students in each of the three grades (pgs. 9-10). Between treatment and control groups, differential attrition rates were 2.9% for kindergarten students, 1.1% for Grade 1 students, and 4% (for the oral fluency outcome) and 5.7% (for the reading comprehension outcome) for Grade 2 cohort 1 and 2 students. Finally, students followed into Grade 3 (cohort 1 of initial Grade 2 sample) had a 0.2% differential attrition rate between conditions (Exhibit/Table 5). No significance tests were reported, and the authors noted that the combination of overall attrition and condition differences did not indicate potential bias according to What Works Clearinghouse standards. Note that the authors' reports of attrition and differential attrition rates do not include those few kindergarten (n=5) and Grade 1 (n=4) students who attrited between randomization and pretest.

In addition, the tests for baseline equivalence in the analysis samples suggested that differential attrition was not a serious problem but lacked a direct comparison to baseline equivalence for the randomized sample.

Posttest:

For all seven short-term (pretest to end-of-school-year posttest) literacy outcomes using separate kindergarten, Grade 1, and Grade 2 samples, students in the intervention group performed better than students in the control group. Specifically, kindergarten intervention students showed higher letter sound fluency (Hedges' g = .83) and decoding fluency (Hedges' g = .53); Grade 1 intervention students scored higher on nonsense word fluency (Hedges' g = .33), oral reading fluency (Hedges' g = .38), and whole word reading (Hedges' g = .24); and Grade 2 intervention students had higher letter sound fluency (Hedges' g = .33) and decoding fluency (Hedges' g = .19).

Long-Term:

Students who received the intervention in Grade 2 scored higher on reading and math in Grade 3 than control group students; however, these effects were not statistically significant. Given that sample was limited to Cohort 1 students, these analyses were somewhat underpowered.