EveryDay Intervention

Study 1

Summary

Robinson et al. (2018) randomly assigned 10,976 households to receive EveryDay Intervention, the EveryDay Intervention with additional support intervention, or a business-as-usual control. Mailings were distributed during the 2015/2016 school year. Attendance data from school districts was used to examine program effects across two different time periods: absences across the entire academic year and absences from the date of the first mailing through the end of the academic year.

Robinson et al. (2018) found that at the end of the academic year, compared to participants in the control condition, participants in the intervention conditions showed significantly greater reductions in:

• Average number of student absences

Evaluation Methodology

Design:

Recruitment: Households were recruited from ten school districts in a diverse county in California. Informed consent mailings were sent to 17,159 households. The study employed a passive consent procedure, allowing parents to opt out of the study at any point during the project by contacting the research team via phone, email or mail. Students from these households who were in kindergarten through fifth grade and in the bottom 60^thpercentile of attendance of participating districts during the prior school year were eligible for participation in the study (all registered kindergarten students were eligible, as they could not be selected based on prior school year attendance data). Students with extreme absences during the prior year and inconsistent records of absences were not eligible for participation. In households with two or more qualifying students, only one student was randomly selected to participate. The initial sample included 10,967 households from ten school districts.

Assignment:

Households were stratified by school, grade, and prior year absences and then randomly assigned to one of two intervention conditions (60%) or the control condition (40%). After the first mailing, a second randomization of only the combined intervention group (i.e., households that were in either of the intervention conditions), stratified by the same variables, was performed to assign half to the intervention condition and the other half to the intervention with additional support condition. The complete randomization process resulted in 3,306 households in the intervention condition, 3,272 households in the intervention with additional support condition, and 4,388 households in the control condition.

Attrition:

Six mailings were sent over the course of six months, beginning in November and ending at the end of the academic year. Absences were measured from the beginning through the end of the academic year and from November (the date of the first mailing) through the end of the academic year. Of the 10,967 assigned households, attendance data was not available for 4% of students, and one student marked absent every day of the year was excluded from analyses, resulting in an overall attrition rate of 4% from baseline to follow-up and an analytic sample of 10,504 households.

In addition, at the end of the school year, the research team conducted phone surveys of eligible households to learn whether the intervention impacted parental beliefs. The phone survey reached 1,710 participating households, 1,599 (93.5%) of which were eligible to participate in the survey. Four hundred seventy-four respondents, or 30% of the eligible participants, completed the entire phone survey.

Sample:

The grade level composition of the sample was 30.65% kindergarten students, 14.25% first grade students, 14.59% second grade students, 13.41% third grade students, 14.05% fourth grade students, and 13.04% fifth grade students. Approximately 37.19% of the sample were of white ethnicity, 17.71% were from a Spanish-speaking household, 31.74% were English language learners (ELL), and 18.38% were considered socioeconomically disadvantaged. The average number of previous year absences was 8.26 days.

Measures:

Attendance data, including excused and unexcused absences, was obtained from each of the respective school districts. Three outcomes were computed for analysis: average number of days absent from the date of the first mailing through the end of the academic year, average number of days absent across the entire academic year, and chronic absenteeism. To directly assess whether the intervention impacted parental beliefs, a subset of parents in eligible households were asked to complete a phone survey at the end of the school year. Additional risk and protective factors assessed during the phone survey included parental perceptions of the number of days their child had been absent during the school year, as well as beliefs about the value of early grade attendance and education.

Analysis:

Students in the two intervention conditions were combined for the primary analyses. Fisher Randomization Tests were used to test whether there was a statistically significant treatment effect on student absences. Linear regression models were used to estimate the average treatment effect of either intervention condition on student absences while adjusting for prior year absences (when available), socioeconomic stats, ELL status, school, and grade. Logit regression models were utilized to examine the average treatment effect on chronic absenteeism, an exploratory outcome.

Intent-to-Treat: All available data were used in analyses.

Outcomes

Implementation Fidelity:

No quantitative tests of implementation fidelity were reported.

Baseline Equivalence:

The percentage of ELL students in the intervention condition (32.55%) was significantly higher than in the control condition (31.02%) at pretest. Otherwise, no significant differences were detected in baseline demographics or outcome variables for the analytic sample.

Differential Attrition:

Students with missing outcome data (4% of the sample) were balanced equally across conditions. No tests of attrition by demographic or outcome variables were reported.

Posttest:

At posttest, students in the intervention conditions were absent for significantly fewer days than students in the control condition. These results were significant when pooling the treatment conditions and when examining them separately, with and without controlling for prior year absences, and when examining all absences accumulated from the beginning of the school year and when only accounting for absences accumulated from the date of the first mailing.

Students in the intervention conditions also showed greater reductions in chronic absenteeism than students in the control condition, but these results were only marginal after controlling for prior year absences. When examining the two treatment groups separately, effects on chronic absenteeism were significant for the enhanced intervention condition (Mailing + Supporter) with and without controlling for prior year absences, but were not significant for the Mailing Only group.

Subgroup analyses indicated that the mailings were more effective among students who identified as ELL, who were significantly more likely to be in the treatment than the control group according to baseline equivalence tests, relative to native English speakers. Mailings were also more effective among students who had the poorest attendance and students from socioeconomically disadvantaged households.

For risk and protective factors (knowledge and attitudes), relative to control participants, parents in either intervention condition showed significantly greater accuracy regarding the number of days of school their child had missed and were more likely to agree with statements about the value of schooling in the early grades and the importance of regular attendance.

Long-Term:

Not examined.

Study 2

Summary

Rogers and Feller (2018) randomly assigned 32,437 households to four conditions, including a control condition and three treatment conditions that varied the information sent to the parent about their child's attendance. Students came from all grades in the Philadelphia school district but were limited to those with more than average absences and less than extremely high absences. Attendance data came from school records covering most of the 2014/2015 school year and from a parent survey of a small subset of the sample.

Rogers and Feller (2018) found that at the end of the academic year, compared to participants in the control condition, participants in the intervention conditions showed significantly greater reductions in:

• Average number of student absences

Evaluation Methodology

Design:

Recruitment: The sample came from students in all grades attending regular schools in the School District of Philadelphia (i.e., not specialized, alternative, charter, or online schools). Of 161,922 students identified in the summer of 2014, 104,088 received parental consent and were also enrolled in the eligible schools. Eight exclusion criteria listed in Table 1 included dropping students who were absent three or fewer days than typical classmates in the same school and grade, who had absences two standard deviations above the mean of the same school and grade, or who were homeless and/or on Individualized Educational Plans. The exclusions reduced the number of eligible students to 40,326. Randomly selecting one focal student per household further reduced the pre-randomization sample to 32,437.

The authors noted that mean baseline absenteeism in the final sample was similar to that of the universe of eligible students, but exclusion of low- and high-absence students reduced the variance.

Assignment: Households were randomly assigned within strata defined by school, grade, and previous year absences to four conditions: 1) parents sent up to five reminders of the importance of attendance and the parental influence on attendance, 2) parents sent up to five reminders with added information on their child's total absences, 3) parents sent up to five reminders with added information on their child's total absences relative to the modal number of absences among classmates, and 4) parents sent no special information on absences (control). The study did not report the randomized sample sizes for each condition, but the sample sizes for the focal students after attrition were: control, n = 6,994; reminder, n = 7,041; total absences, n = 7,037; relative absences, n = 7,008.

Attrition: Data on attendance were obtained from the first mailing in October through the end of the 2014/2015 school year. Table 1 shows a post-randomization sample size of 28,080, or 86.6% of the pre-randomization sample of 32,437. The attrition came from transfers out of the district and missing and incomplete records on attendance. In addition, a survey was completed by 1,268 parents at the end of the school year (but not baseline). The study reported a 23% response rate, but relative to the pre-randomized and post-randomized samples, the completion rates were only 4-5%.

Sample:

The final student sample was 53% African American, 20% Hispanic, 52% female, 28% in high school, and 74% qualifying for free or reduced-price lunches. One-third of all students in Philadelphia lived in households below the poverty line, and 58% of all students scored below basic on the 2014-2015 Pennsylvania System of School Assessment mathematics exams.

Measures:

The primary outcome was the total number of absences (both excused and unexcused) from the date of the first mailing to the end of the school year. Chronic absenteeism was measured as missing 18 or more days of school. The data came from school district administrative records. Secondary outcomes, also from school records, included standardized test scores, course grades, and tardiness. The school district and its teachers and staff were unaware of the treatment group allocation.

The parent survey used single items to measure the 1) belief in the importance of attendance, 2) perceived parent role in reducing absences, 3) parent-child relationship, 4) estimated child total school absences, and 5) estimated child school absences relative to classmates.

Analysis:

The analysis first used the non-parametric joint Fisher randomization test of the null hypothesis of no impact. It then used linear regression to estimate the average treatment effect of each treatment condition, with covariate adjustment for student-level sociodemographics and absences in the previous school year and in the current year prior to randomization, as well as the student's school and grade as fixed effects. Tests used the Holm method to correct for multiple tests across the four conditions (supplement p. 17). Analyses of the survey outcomes also used linear regression with the same controls but no measures of the baseline outcomes.

Intent-to-Treat: The analysis used all available data, regardless of the mailings received, and dropped only those transferring out of the district or having missing or incomplete data.

Outcomes

Implementation Fidelity:

On average, the treatment households received 4.2 mailings over the school year. The survey showed that 57% of the three treatment conditions recalled receiving the treatments compared with 26% in the control group (p < .001).

Baseline Equivalence:

Tests in supplementary Table S2 used the analysis sample to show no significant differences across the four conditions in nine baseline measures, including prior absences, sociodemographic characteristics, and low English proficiency. No baseline equivalence tests for the randomized sample were reported.

Differential Attrition:

For the attendance data, the non-significant tests for baseline equivalence using the analysis sample suggested no systematic attrition bias. In addition, post-randomization attrition rates and percentages of movers did not differ significantly across conditions (see page 341 and supplementary page Table S6), and the authors noted (supplementary Table S7) that when using inverse probability weights the "impacts are nearly identical under a missing at random assumption" (p. 341). For the survey results, tests showed no significant differences between completers and non-completers in nine tests (supplementary Table S8). On request from Blueprints for a more detailed analysis of differential attrition, the authors responded that the standard tests we recommended were cumbersome for studies with four conditions, and that the estimates with inverse probability weights used all the baseline covariates and provided strong evidence that differential attrition was not a problem.

Posttest:

Assignment to the treatment conditions significantly reduced student absences relative to the control group (joint Fisher randomization test p < .001). According to supplementary Table S13, the reminder condition, the total absences condition, and the relative absences condition all had significantly fewer absences than the control group. On average compared to the control group, the reminder group had .62 fewer absent days, the total absences group had 1.1 fewer absent days, and the relative absences group had 1.1 fewer absent days. Similar results emerged for chronic absenteeism and with various sensitivity tests reported in supplementary tables. Treatment effects in the total absences condition and the relative absences condition did not significantly differ from one another, but both were significantly higher than the reminder condition.

Tests for secondary outcomes found no significant effect of the treatment on end-of-the-year standardized test scores, tardy rates, or grades.

Among siblings in study households of focal students, the total absences and relative absences conditions, but not the reminder condition, had significantly fewer absences than the control condition.

Moderation tests found no differences in program effects by gender, race, grade, or prior absenteeism, but they did find significantly larger effects during the week after mailing than two weeks after.

The parent survey showed that the conditions did not differ significantly in parent reports of the importance of absences, their perceived role in reducing absences, or their relationship with the child. For the measure of parent reports of their child's total number of absences, the survey showed significantly more accurate responses in the total absences and relative absences treatment conditions. For the measure of parent reports of their child's relative number of absences, the relative absences condition was significantly more accurate than the other conditions.

Long-Term:

Not examined.

Study 3

The study tested a variation on the program that used a single letter with simplified language and emphasized parental efficacy and highlighted the negative incremental effects of absences.

Summary

Lasky-Fink et al. (2021) used a randomized controlled trial to examine 152,047 truant students from a large urban school district. The students were randomly assigned to seven conditions, with the parents of students in each condition receiving a different type of letter about their child's absences. The number of absences over the next four to nine weeks served as the primary outcome.

Lasky-Fink et al. (2021) found that, relative to a standard treatment control group, three of the six intervention groups had significantly lower:

• School absences over a period of four to six weeks

Evaluation Methodology

Design:

Recruitment: The sample came from three cohorts of K-12 students in a large urban school district who were truant between September and November 2015, newly truant as of December 2015, or newly truant as of January 2016. The full sample consisted of 152,047 students, about 30% of the students in the district, who had missed at least 30 minutes of school without a valid excuse on three occasions. The two exclusions (as specified in the pre-registered analysis plan) included 1) students in households that received more than one treatment assignment in a single randomization round due to inconsistencies and inaccuracies in address data, and 2) students who were randomized after a sibling had been previously assigned (e.g., student A received a truancy notification in round 1 and her sibling, student B, received a notice in round 2).

Assignment: As mandated by the state law, parents of truant students receive a letter. For the study, the 152,047 students were randomly assigned at the household level to one of seven conditions based on different types of letters:

A) Control (standard letter, n = 38,005),

B) Simplified (reworded standard letter, n = 18,963),

C) Efficacy (B plus emphasis on parent action, n = 18,957),

D) Add-Up (B and C plus emphasis on the harm of truancy, n = 19,125),

E) Add-Up + Superintendent (D plus signed by superintendent, n = 18,998),

F) Add-Up + Tips (D plus tips to improve attendance, n = 19,018),

G) Benefits  (B and C plus emphasis on benefits of attendance, n = 18,981).

The six intervention conditions targeted four known behavioral barriers to parental engagement: (1) limited attention; (2) low literacy; (3) feelings of inefficacy; and (4) the common misbelief that a small number of absences is inconsequential. Condition B targeted the first barrier; condition C targeted the first and second; and Conditions D, E, F - the "cumulative conditions" - targeted all four behavioral barriers.

Within each cohort, random assignment took place at the household level and was stratified by grade level, quartile of previous truancy count, and an indicator for Black/African-American students. All students who shared an address were considered to be part of the same household, and all students in a household in a given randomization cohort were assigned to the same treatment condition. To increase the power to detect effects relative to the Standard Notice, 25% of each cohort was assigned to the control condition.

Assessments/Attrition: The assessment period varied across the three cohorts, but ranged from four to nine weeks: cohort 1 from November 1, 2015, to December 8, 2015, cohort 2 from December 10, 2015, to January 20, 2016, and cohort 3 from January 22, 2016, to February 9, 2016. Attrition came from students without end-of-the-year data who were assumed to have moved (2%), students in households that were inadvertently randomized to different conditions in the same randomization cohort due to address discrepancies (3%), and students who were randomized in a subsequent round separately from their sibling (9%). The final analytic sample was 131,312 (86% completion rate).

Sample:

Reflecting overall district demographics, approximately 83% of the sample qualified for free or reduced-price lunch, 12% were Black or African- American, and approximately 50% were Spanish-speaking. On average, students had five unexcused absences prior to randomization.

Measures:

The primary outcome was the total number of absences, both excused and unexcused, as accumulated between each truancy notification mailing and the specified end date. The measure did not count half-day absences but counted partial-day absences in which attendance data for some periods were missing.

Analysis:

The analysis used OLS regressions for the number of absences and the log-transformed number of absences. Standard errors were clustered at the household level, and the models controlled for student-level demographic indicators, school level and type (e.g., magnet school, alternative school), language of truancy notification, randomization cohort, student grade level, and a continuous measure of pre-treatment truancy counts.

Intent-to-Treat: The analysis used all available data, excepting the pre-registered exclusion of households with siblings assigned to different conditions or assigned in different cohorts.

Outcomes

Implementation Fidelity:

The format of the letters meant the content and delivery were standardized.

Baseline Equivalence:

The authors stated that all eight covariates - free and reduced lunch, limited English proficiency, Black/African-American, truancy count, school type, language, cohort, and grade level - were balanced across treatment condition in both the randomized sample and analysis sample. Table S2 of the online supplement lists the baseline means for the analysis sample, which show no significant differences across conditions.

Differential Attrition:

The authors stated that overall attrition rates did not differ across conditions but provided no other information.

Posttest:

Three of the six intervention groups had significantly lower absences than the control group: Add-Up (D), Add-Up + Superintendent (E), and Add-Up + Tips (F). All three used simplified language, emphasized parental efficacy, and highlighted the negative incremental effects of absences. Each of these conditions reduced absences by about 2% in the month after receiving the notice or by approximately 0.07 days from the Standard Notice mean of 3.5 absences. The drop translated into a standardized effect size of .02. The other three intervention conditions - Simplified (B), Efficacy (C), and Benefits (G) - did not significantly reduce absences relative to the Standard Notice.

All results were robust to removing outliers and to a negative binomial specification. Effects were stronger for high school students than younger students but not significantly so. However, the effects came largely in the first 10 days after the mailing.

Long-Term:

Not examined.