Early Literacy and Learning Model

Study 1

Summary

PCERC (2008) examined 28 preschool classrooms and 244 children. Equal numbers of classrooms were assigned to the treatment and waitlist control conditions; however, as part of a pilot study (see Study 2), the treatment classrooms were randomized a year earlier than the control classrooms. Data were collected at baseline, posttest (end of preschool), and follow-up (end of kindergarten), and included measures of literacy and math skills, behavior, and teacher-child interaction.

PCERC (2008) found:

• A delayed effect on vocabulary that showed up at the end of kindergarten (1 year post-intervention)

Evaluation Methodology

Design

The Early Literacy and Learning Model (ELLM) was evaluated in a randomized controlled trial the included classrooms from 28 preschools located in three geographic areas of Florida. The sampled preschools included full-day Head Start, subsidized, faith based, and public school-based early intervention programs. Randomization was done during an earlier national pilot study (2002-2003). However, two preschools dropped out during the pilot year and were replaced for the national study year (2003-2004). A total of 30 classrooms and teachers were randomized, but two dropped out leaving 28 classrooms and teachers for the analysis. The intervention teachers were randomized in 2002 and participated in the pilot study (Study 2) as well as the current study discussed here, while a new pool of classrooms was recruited in 2003 and teachers were randomized to the control condition from this pool.

A total of 297 children and 294 parents were recruited for participation in the national study. The final sample included 244 children (137 treatment, 107 control) and 243 parents. Data were collected on 243 children and 204 parents at the time of the baseline data collection. In the follow-up year of the national evaluation, the sample of schools went from 28 preschools to 119 schools with kindergarten classrooms. The sample of classrooms went from 28 preschool classrooms to 175 kindergarten classrooms. The kindergarten sample included 237 children and 236 parents from original sample of participants. Follow up data were collected on 218 children and 177 parents.

Appendix B (page B-9) reports on completion rates at pretest, posttest (end of preschool), and follow-up (end of kindergarten). For children, the rates were 100%, 92%, and 89%. It is not clear to what extent the national research team attempted to collect post-test and/or follow up data from children who left the study prematurely.

Regarding the counterfactual condition, a number of curricula were used in the control classrooms to include Creative Curriculum (Dodge, Colker and Heroman 2002), & Beyond Centers and Circletime (Phelps 2002), High Reach Learning Pre-K (High Reach Learning 1997) and High/Scope (Hohmann and Weikart 2002).

Sample

The research team recruited preschool programs from three distinct geographic locations within the state. The research team first identified elementary school neighborhoods in each geographic location (Counties A, B, and C) with low-performing schools. Using the Florida Department of Education's school grading report card system the research team identified grade D and F elementary schools in each of the three counties. It was expected that children from the preschool programs in these low-performing elementary school neighborhoods would transition into these grade D and F elementary schools during the kindergarten year of the study. As such, preschool programs within the low performing elementary school neighborhoods were randomly selected for inclusion in the sampling pool of preschool programs.

The average age of children was 4.6 years at the time of the baseline data collection and half (50%) was male. The overall sample was primarily African American (71%) with smaller percentages of White (14%) and Hispanic (8%) children. Approximately 13% of the children were reported by their parents as having a disability.

The average age of the primary caregiver (usually the mother) was 31 years. Almost half (40%) of the primary caregivers were never married; 37% were married at the time of the fall assessment data collection. More than one-third of the primary caregivers reported having had some college (36%) or had graduated from college (6%); 37% had a high school diploma or GED; and 22% had not finished high school. More than half (54%) of the primary caregivers were employed full-time; 11% were employed part-time; and 33% were unemployed. At baseline, a higher percentage of parents in the treatment group had completed some post-high school education relative to those assigned to the control group (41% vs. 29%, p < .01).

There were 28 teachers who participated in the national preschool year intervention study. All were female. The majority identified themselves as African American (64%) or White (21%). The preschool teachers had on average 11 years of teaching experience, with an average of seven years teaching preschool. Fifty percent of the teachers had a high school diploma or GED and 21% had a bachelor's degree. Many of the teachers reported having a state-awarded preschool certificate (52%); a teaching license or certificate (46%); or a Child Development Associate (CDA) credential (46%). Eighteen percent reported having no teacher certification credentials. At baseline, teachers in the treatment group had more years of experience teaching in a preschool setting relative to those assigned to the control group (9 years vs. 4 years, p < .01).

Measures

Data were collected by the national research team from parents, teachers, and schools and through direct assessments of the children. No reliability or validity information was provided on the 23 measures used; however, most are commonly used measures in educational intervention research.

Mathematics

• Woodcock Johnson (WJ) Applied Problems
• Child Math Assessment-Abbreviated (CMA-A) Composite Score
• Shape Composition

Reading

• Test of Early Reading Ability (TERA-3)
• Woodcock Johnson (WJ) Letter Word Identification
• Woodcock Johnson (WJ) Spelling

Phonological Awareness

• Preschool Comprehensive Test of Phonological and Print Processing (Pre-CTOPPP; Elision subtest) (preschool only) & Comprehensive Test of Phonological Processing (CTOPP; Elison subset) (kindergarten only)

Language

• Peabody Picture Vocabulary Test (PPVT)
• Test of Language (TOLD) Grammatical Understanding subtest

Behavior

• Social Skills Rating System (SSRS) Social Skills scale
• SSRS Problem Behaviors scale
• Preschool Learning Behaviors Scale (PLBS) (preschool only) & Learning Behaviors Scale (LBS) (kindergarten only)

Overall classroom environment (baseline & posttest only)

• Early Childhood Environment Rating Scale-Revised (ECERS-R)

Teacher-child interactions (baseline & posttest only)

• Arnett Detachment scale
• Arnett Harshness scale
• Arnett Permissiveness scale
• Arnett Positive Interactions scale

Classroom Instruction (posttest only)

• Early literacy (Teacher Book Reading Scale (TBRS) Print and Letter Knowledge)
• Early literacy (TBRS Written Expression scale)
• Phonological awareness (TBRS Phonological Awareness scale)
• Language (TBRS Book Reading)
• Language (TBRS Oral Language scales)
• Early mathematics (TBRS Math Concepts scale)

Analysis

Analysis was conducted at the individual child-level (for child specific measures) and the classroom-level (for teacher specific measures). Randomization was done at the preschool classroom level. Appendix B (page B-10) describes the use of hierarchical linear models to account for nesting of children within classrooms. Appendix B (page B-8) also states that the analyses used an intent-to-treat logic by including all randomized subjects, even if they were lost to follow-up.

The unadjusted mean scores, adjusted mean difference scores with standard errors, and effect sizes were reported. Effect sizes were calculated using repeat measures (RM) analysis or analysis of covariance (ANCOVA) depending on the data available at each time point.

Adjustments of child-level measures included the following covariates: (a) child's age, (b) gender, (c) race/ethnicity, (d) disability status (as reported by parent), and (e) mother's education. For all analysis of classroom-level measures adjustments included the following covariates: (a) teacher has a BA degree, (b) previous teaching experience, (c) child/adult ratio in classroom, (d) average class size, (e) city size, and (f) geographic site.

RM analysis was used to detect effects on the phonological awareness and social behavioral measures at baseline and post-test. RM analysis was also used to detect effects on the three mathematics outcomes, the three reading outcomes, the overall classroom environment and teacher-child interactions at all three data collection time points.

ANCOVAs were used to assess the long-term effects on phonological awareness and the three social behavioral measures; the covariates, for which, included the baseline score of the measure plus those listed above. ANCOVAs were also conducted for the measures of classroom instruction using the same covariates as listed above; however, classroom instruction data was only collected at post-test, thus baseline data was not included as a covariate.

OUTCOMES

Implementation Fidelity

The research team collected videotaped data to measure the fidelity of ELLM curriculum implementation. Teachers were videotaped twice during the preschool year. The videotapes were segmented and coded to capture the presence or absence of the ELLM critical elements. Scores ranged from 0 to 147. A score of 118 or more was considered high implementation fidelity, 89 to 117 as medium fidelity, and 88 or below as low implementation fidelity.

Even though 11 of the 14 teachers were in their second year of implementing the ELLM, none were rated as high implementers. The intervention teachers were all rated as low to medium implementers. All but one of the control teachers were rated as low implementers (the one exception was rated as a medium implementer). Overall the intervention teachers were given a global 'medium' rating on implementation fidelity.

Baseline Equivalence

At baseline, the groups were equivalent at on all measured outcomes (as listed below), on all child demographics and on most parent and teacher demographics. A higher percentage of parents in the treatment group had completed some post-high school education relative to those assigned to the control group (41% vs. 29%, p < .01). Additionally, teachers in the treatment group had more years of experience teaching in a preschool setting relative to those assigned to the control group (9 years vs. 4 years, p < .01).

Differential Attrition

One control and one treatment preschool dropped out of the study during the preschool year. There is no discussion of how these two preschools differed from the 28 who remained. For individuals, Appendix B (page B-8) states in general that there was no evidence of different rates of attrition across experimental and control groups but does not present specific analyses of differential attrition for this study.

Posttest

Of the 23 measures assessed at post-test (end of pre-k), none showed a statistically significant effect of the program the ELLM children's outcomes relative to the children in the control condition.

Long-term

At the long-term follow-up (1 year after the program ended i.e. end of K), the two child-level measures of language development showed statistically significant effects. They had small to medium effect sizes (PPVT: ES = .34, p < .05; TOLD Grammatical Understanding: ES = .44, p < .05). The other 10 outcome measures were not significantly influenced by the intervention.

Study 2

Summary

Cosgrove et al. (2006) conducted a cluster-randomized trial with 48 classrooms and 466 preschool children, in which classrooms (one per school) were assigned to either the intervention or a waitlist control condition. Measures of alphabet recognition and early literacy skills were collected at baseline and posttest.

Cosgrove et al. (2006), found that in comparison to controls, students in the intervention group showed:

• Greater recognition of letters
• Better emerging literacy skills

Evaluation Methodology

Design

As described above, low-performing public elementary schools housing at least one early intervention pre-kindergarten class were identified in each of the three locations and randomly assigned to either ELLM or wait-list control status. Additionally, two Head Start and two subsidized sites were randomly selected in the neighborhood of each elementary school. One class from each site was randomly selected to participate. Treatment group classes implemented the Early Literacy and Learning Model curriculum in conjunction with existing curriculum. Control group classes only implemented existing curriculum. (Prevalent existing curricula were Creative Curriculum, High/Scope, and High Reach.) The intervention lasted for a year with measures applied at the start and end of the year.

Sample

Four hundred sixty-six  four-year-old children attending preschools in low-income neighborhoods served as the study sample for this investigation. These children were drawn from 48 pre-school classes, from three different geographic locations in Florida. These three locations represented differing degrees of urbanicity. The children were 12% White, 71% Black, 8% Hispanic, and 9% of other racial background.

Measures

Data were collected on the children's ability to recognize the 52 upper- and lowercase letters of the alphabet, and their early literacy abilities were measured by the Test of Early Reading Ability-Third Edition, Form A (TERA-3).

The TERA-3 is composed of three scales measuring unique but related early literacy skills: 1) the Alphabet scale measures graphophomenic knowledge; 2) the Conventions of Print scale measures knowledge of conventions of English print; and 3) the Meaning scale measures ability to comprehend meaning of print. Cronbach's Alpha coefficients of internal consistency for 4-year-old children for the Reading Quotient and the Alphabet, Conventions, and Meaning subtests are .97, .94, .88, and .94 respectively. Alpha coefficients for the 5-year-old children were comparable.

The Alphabet Letter Recognition Inventory (ALRI) is a locally developed inventory of children's ability to recognize the upper- and lowercase letters of the alphabet. Trained assessors presented uppercase letter flashcards, arranged in a fixed non-alphabetic order, to each child. The child was asked to name the letter. Following presentation of the uppercase letters, lowercase letter flashcards were presented in a similar fashion. The children's responses were recorded on scannable forms and computer scored.

Analysis

Hierarchical linear modeling (HLM) was used to test the statistical significance of these observed differences because children experienced ELLM together in classes rather than in one-on-one settings. Child-level variables included in the analyses were the TERA-3 pretest standardized scores, the ALRI pretest scores, the child's age in months on September 1, 2002, and gender. All continuous variables were grand-mean centered. Class-level variables included in the analyses were class assignment as ELLM or wait-list control, urbanicity, and educational attainment of the teacher (coded as either having completed a bachelor's degree or not).

Baseline Equivalence and Differential Attrition

At baseline, and again, near the end of the school year, children were assessed on their emerging literacy abilities and on their ability to recognize the 52 upper- and lowercase letters of the alphabet. Treatment children did not differ significantly from control children on these measures at baseline.

Attrition occurred at both the site and child levels. One wait-list control and one ELLM classroom in the largest city withdrew from the study and were not included in the analysis. In the 48 remaining classrooms, child-level attrition occurred, and 21% of the remaining children with pretest scores were unavailable for posttest. Comparison of the TERA-3 mean pretest scores of children with both pretest and posttest scores to those of the children with only pretest scores provided no evidence of systematic attrition.

Posttest

Examination of the TERA-3 and ALRI posttest scores indicated that those of the ELLM children were higher than those of the wait-list control children on all measures.

The child-level variables were statistically significant predictors in all of the analyses except for the child's gender in the Analysis of the Meaning scale. In all cases, older children scored at lower posttest levels than younger children, and, where gender was significant, boys achieved at lower levels than girls. These findings were the same for children in the ELLM and wait-list control classrooms.

At the class level, only assignment to ELLM or wait-list control classes was a statistically significant predictor in all analyses. The educational attainment of the classroom teacher was a statistically significant predictor only in the analysis of the Conventions of Print scale, and for this analysis, children who were taught by bachelor-degreed teachers achieved at higher levels in both the ELLM and wait-list control classrooms.

Analyses indicated that ELLM was more effective than traditional approaches in raising the emergent literacy achievement of children. As measured by all TERA-3 scales, the children experiencing the ELLM literacy curriculum and instruction support system achieved higher adjusted mean posttest levels of emergent literacy skills than children in the wait-list control classrooms. With the exception of the differences in the TERA-3 Conventions of Print scale adjusted mean posttest scores, resulting effect sizes were 25 percent of a standard deviation or larger (ARLI ES=.25, TERA-3 Reading ES=.28, TERA-3 Alphabet=.28, TERA-3 Conventions of Print=.17, and TERA-3 Meaning ES= .29).