A college advising program designed to support low-income and first-generation students during the college application process and throughout college enrollment.
Blueprints: Promising
Crime Solutions: Promising
OJJDP Model Programs: Promising
Social Programs that Work:Top Tier
What Works Clearinghouse: Meets Standards Without Reservations - Potentially Positive Effect
Steve Colón, Chief Executive Officer
Erin Harris, Director of Data and Evaluation
eharris@bottomline.org
Bottom Line
50 Milk St., 16th Floor
Boston, MA 02109
857-415-4810
www.bottomline.org
Andrew Barr, Evaluator
Department of Economics, Texas AM University
Bottom Line provides individualized college advising to high school students starting at the end of their junior year. Eligibility criteria for the program includes: 1) a high school GPA of at least 2.3; 2) family income below 200 percent of the poverty line; and 3) being in the first generation in their family to go to college in the United States. Students are initially admitted into the "Access" program, which provides students with college and financial aid application support starting the summer before their senior year in high school through the summer after high school graduation. During this time, advisors help students identify well-matched colleges and then complete and submit college applications. The Bottom Line "Access" program also helps students evaluate the affordability of different postsecondary options, and then advisors actually help students apply for financial aid. Barr & Castleman (2017) reported that Bottom Line operates in several cities in 3 states (Massachusetts, New York, and Illinois). As reported in Barr & Castleman (2017), a unique component of the program is that counselors encourage students to attend a set of target colleges and universities within New York and Massachusetts that the program has identified as providing students with an optimal combination of quality and affordability. Students who enroll at one of the target institutions (approximately 50 percent of the students in Barr & Castleman, 2017) are eligible for the Bottom Line "Success" program, in which individualized and campus-based support is provided for up to six years following high school. Two important contrasts between the Bottom Line model and other advising programs are: 1) Bottom Line employs professional advisors (all have a minimum of a college degree) who work full-time; and 2) the intensity of the model - the average student spends 10-15 hours with their advisor before transitioning to college. Advisors have an average caseload of 50-60 students and meet with each student for an hour every three or four weeks during senior year, at Bottom Line's office in each community.
Bottom Line provides individualized college advising to high school students starting at the end of their junior year. Eligibility criteria for the program includes: 1) a high school GPA of at least 2.3; 2) family income below 200 percent of the poverty line; and 3) being in the first generation in their family to go to college in the United States. Students are initially admitted into the "Access" program, which provides students with college and financial aid application support starting the summer before their senior year in high school through the summer after high school graduation. During this time, advisors help students identify and apply to colleges that are a strong match in terms of affordability, competitiveness and fit. The Bottom Line "Access" program advisors also help students apply for financial aid.
Barr & Castleman (2017) reported that Bottom Line operates in three cities in 3 states (Boston, Massachusetts, New York, NY, and Chicago, Illinois). As reported, a unique component of the program is that counselors encourage students to attend a set of target colleges and universities typically in and around the cities where they work that the program has identified as providing students with an optimal combination of quality and affordability. Students who enroll at one of the target institutions (approximately 50 percent of the students in Barr & Castleman, 2017) are eligible for the Bottom Line "Success" program, in which individualized and campus-based support is provided for up to six years following high school.
Through the "Success" program, advisors provide individualized and campus-based support to college students throughout their college enrollment for up to 6 years, with the goal that students will graduate with a Bachelor's degree in 6 years or less. Bottom Line provides ongoing advising during the summer after high school to help students navigate and complete required pre-matriculation tasks, such as attending orientation, completing placement tests, and setting up a tuition payment plan. Campus-based advisors at each target institution continue to meet regularly with students once they have matriculated into college. First-year students meet with advisors an average of 3 to 4 times per semester, and older students meet with an advisor an average of twice a semester. Advisors in the Bottom Line "Success" program provide a combination of academic support (e.g., course selection and making use of advising and tutoring services) and social support (e.g., helping students adjust to a new environment, getting involved with activities and student groups), and they also advise students on how to balance academic, work, social, and family commitments while in college.
Two important contrasts between the Bottom Line model and other advising programs are: 1) Bottom Line employs professional advisors (all have a minimum of a college degree) who work full-time; and 2) the intensity of the model - the average student spends 10-15 hours with their advisor before transitioning to college. Advisors have an average caseload of 50-60 students and meet with each student for an hour every three or four weeks during senior year, at Bottom Line's office in each community.
Primary Evidence Base for Certification
Study 1
Barr and Castleman (2017, 2021) found that, as compared to control students, treatment students showed higher rates in:
Primary Evidence Base for Certification
Of the two studies Blueprints has reviewed, one (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness). The study was done by the developer.
Study 1
Barr and Castleman (2017, 2021) conducted a randomized controlled trial to evaluate the effect of Bottom Line on students' college enrollment (overall) and enrollment into 4-year and 2-year colleges. The sample included 2,422 students from 2 cohorts (high school graduating classes of 2015 and 2016) across 2 regions (Boston & Worcester, MA and New York City, NY). Outcome measures were assessed using college enrollment data from the National Student Clearinghouse.
Study 1
Barr, A., & Castleman, B. (2017). The Bottom Line on college counseling. Working Paper.
Barr, A. C., & Castleman, B. L. (2021). The Bottom Line on College Advising: Large increases in degree attainment (EdWorkingPaper 21-481). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/xdsa-5e22.
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
In Study 1, Barr & Castleman (2017) tested for subgroup effects by race, ethnicity, and gender and found equal benefits across groups. However, Barr & Castleman (2021) tested for subgroup effects with subsample analyses and found benefits for Asian Americans, African Americans, Hispanics, males, and females but did not report tests that compared the subgroups to one another.
Sample demographics including race, ethnicity, and gender for Blueprints-certified studies:
In the Study 1 sample (Barr & Castleman, 2017, 2021), the students were 70% female, 31% Black, 29% Hispanic, 24% Asian, and 9% of other races.
Bottom Line, founded in 1997, began programming in Massachusetts (serving Boston and Worcester) and has since replicated its model in Brooklyn, New York and Chicago, Illinois. While Bottom Line does not sell materials or a training program, it readily engages with organizations and/or community leaders interested in similar and accurate replications. Bottom Line operates with a centralized organizational model which ensures the integrity and consistency of the program. In addition to a National Program Team that leads program design, training, and evaluation, centralized support services are leveraged to oversee the organization's finance, operations, and human resources functions. Regions are led by an Executive Director, who is supported by program and development leadership.
In terms of training, Bottom Line Advisors go through an intensive initial training period upon hire. Following the on-boarding, training is provided on each successive portion of the Bottom Line curriculum and materials. An annual new employee orientation is held each summer which provides both behavioral and technical trainings, including program-specific trainings for all new staff. The National Human Resource department also leads trainings for all staff on core competencies; performance management; Diversity, Equity, and Inclusion; and other topics as needed. These trainings occur in person when possible, or virtually as needed. In addition, regions coordinate their own trainings on topics of interest to their specific staff. Ongoing training and professional development are key factors contributing to Bottom Line's college success rates.
Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.
For organizations and/or communities interested in replication of the Bottom Line model, see Program Information Contact listed on the Fact Sheet.
No information is available
No information is available
No information is available
No information is available
No information is available
No information is available
No information is available
No information is available
No information is available
No information is available
No information is available
Andrew Barr, EvaluatorDepartment of Economics, Texas AM Universityabarr@tamu.edu
A college advising program designed to support low-income and first-generation students during the college application process and throughout college enrollment.
Low-income and first-generation high school students
Subgroup differences in program effects by race, ethnicity, or gender (coded in binary terms as male/female) or program effects for a sample of a specific racial, ethnic, or gender group:
In Study 1, Barr & Castleman (2017) tested for subgroup effects by race, ethnicity, and gender and found equal benefits across groups. However, Barr & Castleman (2021) tested for subgroup effects with subsample analyses and found benefits for Asian Americans, African Americans, Hispanics, males, and females but did not report tests that compared the subgroups to one another.
Sample demographics including race, ethnicity, and gender for Blueprints-certified studies:
In the Study 1 sample (Barr & Castleman, 2017, 2021), the students were 70% female, 31% Black, 29% Hispanic, 24% Asian, and 9% of other races.
*Risk/Protective Factor was significantly impacted by the program
Bottom Line provides individualized college advising to high school students starting at the end of their junior year. Eligibility criteria for the program includes: 1) a high school GPA of at least 2.3; 2) family income below 200 percent of the poverty line; and 3) being in the first generation in their family to go to college in the United States. Students are initially admitted into the "Access" program, which provides students with college and financial aid application support starting the summer before their senior year in high school through the summer after high school graduation. During this time, advisors help students identify well-matched colleges and then complete and submit college applications. The Bottom Line "Access" program also helps students evaluate the affordability of different postsecondary options, and then advisors actually help students apply for financial aid. Barr & Castleman (2017) reported that Bottom Line operates in several cities in 3 states (Massachusetts, New York, and Illinois). As reported in Barr & Castleman (2017), a unique component of the program is that counselors encourage students to attend a set of target colleges and universities within New York and Massachusetts that the program has identified as providing students with an optimal combination of quality and affordability. Students who enroll at one of the target institutions (approximately 50 percent of the students in Barr & Castleman, 2017) are eligible for the Bottom Line "Success" program, in which individualized and campus-based support is provided for up to six years following high school. Two important contrasts between the Bottom Line model and other advising programs are: 1) Bottom Line employs professional advisors (all have a minimum of a college degree) who work full-time; and 2) the intensity of the model - the average student spends 10-15 hours with their advisor before transitioning to college. Advisors have an average caseload of 50-60 students and meet with each student for an hour every three or four weeks during senior year, at Bottom Line's office in each community.
Bottom Line provides individualized college advising to high school students starting at the end of their junior year. Eligibility criteria for the program includes: 1) a high school GPA of at least 2.3; 2) family income below 200 percent of the poverty line; and 3) being in the first generation in their family to go to college in the United States. Students are initially admitted into the "Access" program, which provides students with college and financial aid application support starting the summer before their senior year in high school through the summer after high school graduation. During this time, advisors help students identify and apply to colleges that are a strong match in terms of affordability, competitiveness and fit. The Bottom Line "Access" program advisors also help students apply for financial aid.
Barr & Castleman (2017) reported that Bottom Line operates in three cities in 3 states (Boston, Massachusetts, New York, NY, and Chicago, Illinois). As reported, a unique component of the program is that counselors encourage students to attend a set of target colleges and universities typically in and around the cities where they work that the program has identified as providing students with an optimal combination of quality and affordability. Students who enroll at one of the target institutions (approximately 50 percent of the students in Barr & Castleman, 2017) are eligible for the Bottom Line "Success" program, in which individualized and campus-based support is provided for up to six years following high school.
Through the "Success" program, advisors provide individualized and campus-based support to college students throughout their college enrollment for up to 6 years, with the goal that students will graduate with a Bachelor's degree in 6 years or less. Bottom Line provides ongoing advising during the summer after high school to help students navigate and complete required pre-matriculation tasks, such as attending orientation, completing placement tests, and setting up a tuition payment plan. Campus-based advisors at each target institution continue to meet regularly with students once they have matriculated into college. First-year students meet with advisors an average of 3 to 4 times per semester, and older students meet with an advisor an average of twice a semester. Advisors in the Bottom Line "Success" program provide a combination of academic support (e.g., course selection and making use of advising and tutoring services) and social support (e.g., helping students adjust to a new environment, getting involved with activities and student groups), and they also advise students on how to balance academic, work, social, and family commitments while in college.
Two important contrasts between the Bottom Line model and other advising programs are: 1) Bottom Line employs professional advisors (all have a minimum of a college degree) who work full-time; and 2) the intensity of the model - the average student spends 10-15 hours with their advisor before transitioning to college. Advisors have an average caseload of 50-60 students and meet with each student for an hour every three or four weeks during senior year, at Bottom Line's office in each community.
Through personalized one-on-one guidance and support provided by advisors, students can achieve their goals of college entrance and graduation.
Primary Evidence Base for Certification
Of the two studies Blueprints has reviewed, one (Study 1) meets Blueprints evidentiary standards (specificity, evaluation quality, impact, dissemination readiness). The study was done by the developer.
Study 1
Barr and Castleman (2017, 2021) conducted a randomized controlled trial to evaluate the effect of Bottom Line on students' college enrollment (overall) and enrollment into 4-year and 2-year colleges. The sample included 2,422 students from 2 cohorts (high school graduating classes of 2015 and 2016) across 2 regions (Boston & Worcester, MA and New York City, NY). Outcome measures were assessed using college enrollment data from the National Student Clearinghouse.
Primary Evidence Base for Certification
Study 1
Barr and Castleman (2017) found that treatment students were more likely to be enrolled in college than control students, and they were more likely to enroll in 4-year universities and less likely to enroll in 2-year colleges (compared to control students). Treatment students were also significantly more likely to persist in college (i.e., be enrolled in a second year, have more total enrolled semesters, and be continuously enrolled). Barr & Castleman (2021) extended the finding by showing that treatment students were more likely than control students to attain degrees within 5-6 years from a 4-year college and from a high-quality 4-year college, and they were less likely to attain an associate degree from a 2-year college.
Primary Evidence Base for Certification
Study 1
Barr and Castleman (2017, 2021) found that, as compared to control students, treatment students showed higher rates in:
In Study 1, Barr & Castleman (2021) reported from a partial mediation analysis that most of the treatment effect on degree attainment at 4-year colleges resulted from advisors shaping student applications and college choices toward higher-quality institutions. Study 2 (Castleman & Goodman, 2017) did not examine mediating effects.
One study meets Blueprints standards for high-quality methods with strong evidence of program impact (i.e., "certified" by Blueprints): Study 1 (Barr & Castleman, 2017, 2021). The sample for the study included low-income and first-generation high school students. Study 1 took place in Boston and Worcester, MA, and New York City, NY, and compared the treatment group to a services-as-usual control group.
Additional Studies (not certified by Blueprints)
Study 2 (Castleman and Goodman, 2017)
Castleman, B. L., & Goodman, J. (2017). Intensive college counseling and the college enrollment choices of low income students. Education Finance and Policy, 13(1), 19-41.
An early report examined only the first of the two cohorts included in Study 1. Barr & Castleman (2016) found positive program effects on college enrollment, but the results were preliminary and replaced by the larger sample in Barr & Castleman (2017) reported below.
Barr, A. C., & Castleman, B. L. (2016). Advising students to and through college: Experimental evidence from the Bottom Line Advising program. Bottom Line. Available online: https://www.bottomline.org/sites/default/files/Advising%20Students%20To%20and%20Through%20College_web.pdf.
Blueprints: Promising
Crime Solutions: Promising
OJJDP Model Programs: Promising
Social Programs that Work:Top Tier
What Works Clearinghouse: Meets Standards Without Reservations - Potentially Positive Effect
Steve Colón, Chief Executive Officer
Erin Harris, Director of Data and Evaluation
eharris@bottomline.org
Bottom Line
50 Milk St., 16th Floor
Boston, MA 02109
857-415-4810
www.bottomline.org
Certified Barr, A., & Castleman, B. (2017). The Bottom Line on college counseling. Working Paper.
Certified
Barr, A. C., & Castleman, B. L. (2021). The Bottom Line on College Advising: Large increases in degree attainment (EdWorkingPaper 21-481). Retrieved from Annenberg Institute at Brown University: https://doi.org/10.26300/xdsa-5e22.
Barr, A. C., & Castleman, B. L. (2019). Exploring variation in college counselor effectiveness. American Economic Association Papers and Proceedings, 109, 227-231.
Castleman, B. L., & Goodman, J. (2017). Intensive college counseling and the college enrollment choices of low income students. Education Finance and Policy, 13(1), 19-41.
Summary
Barr and Castleman (2017, 2021) conducted a randomized controlled trial to evaluate the effect of Bottom Line on students' college enrollment (overall) and enrollment into 4-year and 2-year colleges. The sample included 2,422 students from 2 cohorts (high school graduating classes of 2015 and 2016) across 2 regions (Boston & Worcester, MA and New York City, NY). Outcome measures were assessed using college enrollment data from the National Student Clearinghouse.
Barr and Castleman (2017, 2021) found that, as compared to control students, treatment students showed higher rates in:
Evaluation Methodology
Design:
Recruitment: Eligibility for the program and study included having a GPA of at least 2.5 and family income below 200% of the poverty line. To recruit participants, staff actively promoted their program through high schools and non-profit partners in three community sites: Boston, New York City, and Worcester, Massachusetts. A total of 2,422 students joined the study in two separate cohorts (the high school graduating classes in 2014 and 2015). The number of high schools from which students attended was not reported.
Assignment: Students were randomized via a lottery system to treatment (n = 1,687) or control (n = 735) across two regions (Boston-Worcester and New York City) and two cohorts (graduating classes of 2014 and 2015). The uneven condition numbers stemmed from "minimum commitments to its funders and community partners of the number of students it had to serve." Intervention students received an offer to participate in the Bottom Line Access program, while control participants did not receive any services from Bottom Line but were able to access college planning guidance and support from other sources. Some treatment students got Bottom Line "Access" and "Success" and some got just Bottom Line "Access," but all were randomized to one treatment group.
Attrition: The study followed participants from high school to two years after high school in Barr & Castleman (2017) and to five years (for the second cohort) or six years (for the first cohort) after high school in Barr & Castleman (2021). Administrative records on college enrollment were collected on 2,422 students, so there was no attrition. However, the most up-to-date measures in both reports (Barr & Castleman, 2017, 2021) were limited to the first cohort (n = 1429), which started earlier; the second cohort (n = 993) started later but otherwise had no missing data. For the two surveys examined in Barr & Castleman (2021), the completion rate was about 56%.
Sample: Sample characteristics were provided by condition (see Table 1 in Barr & Castleman, 2017). Among the treatment students, 70.1% were female, and the racial/ethnic composition of the students was: 32.4% Black, 31.7% Hispanic, 23.7% Asian, and 9.5% other races. Among the control students, 69.7% were female, and in terms of race/ethnicity, students were 30.2% Black, 32.5% Hispanic, 24.6% Asian, and 9.4% other races. Both treatment and control groups had a first-generation student rate of 81.1%. It appeared that students were low income, as their parents' average gross income was $22,520 a year.
In terms of counselors, most counselors were female (75%), with roughly a quarter black and a quarter Hispanic. The median counselor age was 26 years.
Measures:
Barr & Castleman (2017): Outcome data came from the National Student Clearinghouse, which provided students' term-level college enrollment data into 2-year and 4-year colleges. College entrance (as in the period of time between high school graduation and entrance into college), however, was not defined, although one cohort (2015) had up to 2 years to potentially enroll in college after high school graduation, while the other (2016) had up to 1 year of potential opportunity to enroll in college. College persistence was measured as enrollment in a second year, total enrolled semesters, and continuous enrollment. College quality was measured as enrollment in the student's target college and enrollment in a college with a high graduation rate, high SAT/ACT entrance scores, and a low default rate.
Outcome data were collected from the National Student Clearinghouse (NSC), "with coverage across 96% of college enrollments in the country" (p. 9). Given that no missing outcome data were reported, it appeared that students without a record in the NSC database were assigned a 0, indicating they did not enroll in college. This, however, was not explicitly stated.
Barr & Castleman (2021): Along with measures on enrollment and degree attainment from the National Student Clearinghouse, this report examined the quality of the colleges in which the students enrolled. Participants in both cohorts were followed for five years after high school, and participants in the first cohort were followed for six years. College quality was measured by being a designated Bottom Line target college and by graduation rates, loan default rates, post-graduation earnings, and post-graduation intergenerational "social mobility" that was above the median. In addition, two surveys conducted with students during the spring of their senior year in high school and the fall after high school graduation measured several risk and protective factors related to progress toward college and financial milestones (e.g., completing college applications and financial aid forms).
Analysis: Barr and Castleman (2017, 2021) pooled across cohorts to conduct the analysis. Main effects were tested using linear and logistic regression models, controlling for site-by-cohort fixed effects. Other covariates included demographics, family resources and background, academic aptitude (standardized GPA, state standardized test scores), and measures of college guidance resources (whether the student was working with another advising organization). The models estimated robust standard errors.
Intent-to-Treat: All participants were analyzed according to their assigned condition, which is in line with the intent-to-treat protocol.
Outcomes
Implementation Fidelity:
Barr & Castleman (2017) reported data on dosage. Bottom Line counselor interaction data indicated that 97% of the treatment students had at least one interaction with a counselor. In addition, over a 15-month period, counselors interacted with students an average of 13 times, with the majority of these interactions occurring as in-person meetings in the counselor's office.
Barr & Castleman (2019, 2021) further reported that 29 out of 30 advisors had a positive effect on the 4-year enrollment of advisees and that advisor gender or race has no relationship with student success.
Baseline Equivalence:
Tests in Barr & Castleman (2017, 2021) showed significant differences (p < .05) between the treatment and control groups in 2 (out of 19) variables: treatment students were: 1) less likely to be a U.S. citizen, and 2) more likely to have a father who was employed. While the main effects analysis controlled for socio-demographic variables, it was not clear whether these specific variables were included in the model.
Differential Attrition:
The main analysis of college outcomes had no attrition. The attrition rates of the two surveys examined in Barr & Castleman (2021) were roughly equal among treatment and control group students. Tests for baseline equivalence in the analysis sample showed the same two significant differences in 19 tests as for the tests using the full randomized sample (Table A5).
Posttest:
For at least some of the intervention participants, those participating in the Bottom Line Success program for college students, the program was ongoing during the assessment period. Only the measures of degree attainment served as full posttest outcomes.
Barr & Castleman (2017): Results after two years, when the program was ongoing, showed that treatment students were significantly more likely to be enrolled in any college than control students, and the effect was larger when looking at 4-year college enrollment. At the posttest, treatment students were 7% points more likely to enroll in any college than the control students. A larger effect was found on 4-year college enrollment, with a 10%-point increase among treatment students as compared to the control group. In contrast, the program significantly reduced enrollment in 2-year colleges. Additional tests showed that the program significantly increased enrollment in the student's target college and enrollment in colleges with high graduation rates, high SAT/ACT entrance scores, and a low default rate.
Supplemental analyses indicated that treatment effects in college enrollment were larger for Hispanics and individuals with lower high school GPAs.
College persistence was reported for only one cohort (students in the 2014 high school graduating cohort). The results, again during the period when the program was ongoing, indicated that compared to control students, treatment students experienced significantly higher second-year enrollment in any college, second-year enrollment in a 4-year college, total enrolled semesters, and continuous enrollment.
Barr & Castleman (2021): Tests for program main effects replicated the results for enrollment in Barr & Castleman (2017), but the study also presented results on degree attainment over a 5-6-year period. The intervention students had significantly higher degree attainment rates at a 4-year college and significantly higher degree attainment at a college classified as high quality (i.e., having higher graduation rates, loan non-default rates, post-graduation earnings, and post-graduation intergenerational "social mobility" above the median). The program effects on degree attainment at 2-year colleges were negative, as the program diverted students from 2-year to 4-year colleges. Moderation tests found similarly beneficial effects across sites, cohorts, advisors, and student characteristics.
Tests using the survey data to examine the completion of college and financial aid milestones at the early stages of the program found only one significant effect (p < .05): The intervention students completed a larger number of applications.
Additional tests using recently developed causal forest methods found that increases in bachelor's degree attainment were due primarily to shifting students to enroll at higher-quality institutions. Mediation analysis suggested a modest role of ongoing college advising for degree attainment.
Long-Term: Because program students who attended college continued to receive counseling for up to six years after high school, the study did not examine long-term effects.
Summary
Castleman and Goodman (2017) used a regression discontinuity quasi-experimental design to examine 4,992 high school students in Massachusetts who applied to the program in 2010-2014. Conditions were defined as those with a GPA under the threshold of eligibility who did not receive the program counseling and those with a GPA at or above the threshold of eligibility who did receive the program services. For the full sample, the outcome measures included college enrollment in the first year after high school.
Castleman and Goodman (2017) found that, among students with a GPA near the 2.5 threshold for acceptance into the program, those who received the program counseling relative to those who did not were significantly:
Evaluation Methodology
Design:
Recruitment: The sample consisted of 4,992 Bottom Line applicants from the high school classes of 2010 through 2014 who 1) had GPAs between 1.0 and 4.0 and 2) were successfully merged with Massachusetts state educational data. The sample came from two program sites, one in Boston and one in Worcester, MA. All participants were seeking to attend college. A subsample of 2,730 participants consisted of those in the graduating classes of 2010 through 2012, who could be followed for a longer period than those in later classes.
Assignment: The quasi-experimental regression discontinuity design compared those with a GPA below 2.5 who were not eligible for services to those with a GPA of 2.5 and higher who were eligible for services. About 55% of the sample, mostly but not fully from the group with the higher GPAs, actually received the program counseling services. The design assumed that those just below and just above the 2.5 GPA threshold were nearly identical not only in terms of GPA but also in academic skills and other characteristics and therefore differed only in access to the program counseling services.
Assessments/Attrition: The study examined college enrollment in the fall following high school graduation and, for a subgroup, enrollment in the spring of the second and third years following graduation. There was no formal attrition, as the sample included only those who could be matched to administrative databases. About 7% of potential participants were unmatched and excluded from the sample.
Sample:
About 80% of the students were low income, as measured by receipt of subsidized lunches. Most were Black (41%), Hispanic (28%), or Asian (21%), and 68% were female. Over half (54%) spoke a language at home other than English.
Measures:
Data for the study came from administrative records provided by Bottom Line, the Massachusetts Department of Elementary and Secondary Education, and the Integrated Postsecondary Education Data System. Information on students from Bottom Line was matched with the other databases. The Massachusetts data had been merged with National Student Clearinghouse data, which tracks college enrollment throughout the United States and had coverage rates around 95% for recent Massachusetts cohorts. The key measure of GPA, which defined eligibility in the design, came from student self-reports to Bottom Line rather than from administrative records. However, program staff verified eligibility by checking high school transcripts.
The outcome measures included enrollment in two- and four-year colleges. The measures distinguished between encouraged colleges where costs were lower and graduation rates higher and discouraged colleges where costs were higher and graduation rates lower. Additional data measured institutional characteristics of the colleges attended by the students, such as six-year graduation rates and average net prices.
Analysis:
The study presented non-ITT results using fuzzy regression discontinuity models to examine the impact of counseling itself rather than eligibility for counseling based on GPA. The model first used eligibility as an instrument to predict receipt of counseling, and it then examined the impact of predicted counseling on college enrollment. The estimates produced local average treatment effects of receiving counseling that were based on the difference in college outcomes between students just above and just below the GPA threshold. As required by the model, there was a clear discontinuity of receiving counseling at the threshold - those above the threshold had a treatment probability that was 25 percentage points higher than for those below the threshold.
The analysis used linear regression models. Control variables included GPA (as the distance from the 2.5 threshold), eligibility for counseling, and the interaction between the two. The models for college enrollment could not include the baseline outcome but added controls for graduating class, sociodemographic characteristics, and agency site.
Intent-to-Treat: By examining the effect of counseling received rather than eligibility for counseling, the study did not present an ITT analysis. By using eligibility as an instrument, however, the authors aimed to adjust for condition differences resulting from the non-ITT analysis.
Outcomes
Implementation Fidelity:
Not examined.
Baseline Equivalence:
The regression discontinuity design required that students did not systematically manipulate on which side of the GPA threshold they fell in order to qualify for the program. The authors reported that there was no evidence that GPAs just above 2.5 were over-represented relative to GPAs just below 2.5, suggesting no obvious manipulation by students. They also checked for bias by estimating a regression discontinuity model with a "donut hole" that excluded those who had a GPA very close to the 2.5 threshold.
The design also required that student applications to the program not be influenced by knowledge of the GPA threshold and that students just above the threshold be similar to those just below the threshold. The authors reported that nine of ten observable covariates were balanced across the threshold and that the one imbalance was small and likely due to chance (see Appendix Table A.2).
Differential Attrition:
The 7% of students who were unmatched and excluded from the study came from private schools, were missing school information, had common names that could not be uniquely matched, or used misspelled names, nicknames, or other non-legal names. The authors reported that unmatched applicants looked "demographically quite similar to the sample as a whole" and that match rates were unrelated to treatment status.
Posttest:
The non-ITT regression discontinuity results in Table 3 list 28 tests - seven outcomes by four bandwidths around the GPA threshold - and 12 were significant (p < .05). Among students with GPAs near the 2.5 threshold, those who received the program counseling were significantly more likely to enroll in a college encouraged by the program (four of four tests), less likely to enroll in a discouraged college (four of four tests), and more likely to enroll in a lower-cost college (three of four tests) than those who did not receive the counseling. There was mixed evidence for a significant program effect on reducing enrollment in a two-year college (one of four tests) and no significant program effect for enrollment in a four-year college or any college (zero of four tests).
For the subset of older cohorts who could be followed longer, the results in Table 4 show five of 28 tests to be significant (p < .05). Those who received the counseling relative to those who did not were significantly more likely to have been enrolled in all semesters through spring of year two (three of four tests) and be enrolled in a four-year college as of spring of year two (two of four tests). There were no significant effects for year three.
Long-Term:
With the program lasting up to six years of college, there was no long-term follow-up.