Researchers examined the impact of BELL’s middle school summer learning model in three school districts. They used a random assignment research design, the most rigorous type of design for evaluating program effects.

Because admission to the program was determined using random assignment, students in the BELL and the non-BELL groups were comparable with respect to their motivation and ability at the start of the program. This means that any subsequent difference between the two groups with respect to academic outcomes in the fall after participating in the program can be attributed to the impact of the BELL program.

The study was conducted in summer 2012. In a typical summer, BELL fills its limited program slots on a “first-come, first-served” basis. For the purposes of this study, however, random assignment was used to select which students would be admitted to BELL. Schools in these districts identified students who were performing below grade level and encouraged them to apply to the program. In total, 1,032 rising sixth-, seventh-, or eighth-grade students applied to the middle school program and agreed to be part of the study. A lottery-like process was used to determine which students would be invited to participate in the BELL middle school program. 

Data collection and the analysis sample

Information about students’ characteristics at baseline was obtained from the application form for BELL. Schools also provided data on students’ scores on state tests in the spring before program participation. Classroom observations, interviews, and focus groups with staff were conducted in the third and fourth weeks of the program. Attendance data were obtained from BELL.

In the fall after the summer program, students in both groups took a reading achievement test (Group Reading Assessment and Diagnostic Examination, or GRADE) and a math achievement test (Group Mathematics Assessment and Diagnostic Examination, or GMADE), and they completed a survey.

The analysis of impacts is based on 919 students (89 percent of the study sample) for whom fall 2012 achievement and survey data are available.

Outcomes

The study focuses on:

•  Reading achievement test scores
•  Math achievement test scores
• Student engagement in fall 2012, after participating in the program.

Limitations

The study has three main limitations.

• The margin of error around the impact findings is quite large.
• The three school districts in the study were new partnerships for BELL in summer 2012, and they operated voluntary (rather than mandatory) programs; therefore, the findings may not be representative of the effect of BELL’s program in districts that have more experience with it or in districts where student participation is mandatory.
• BELL’s middle school model has changed since summer 2012 — especially with respect to teacher training and the math and literacy curricula; thus, the findings may not be representative of the impact of the model as it now exists. 

Given these limitations, this study cannot provide conclusive evidence of impacts, and its findings may not be generalizable.

Because of random assignment, however, the findings can still be useful for generating preliminary evidence about the potential effects of middle school summer programs and for understanding the environment in which they operate.