Afterschool

Measuring Youth Development

How Out-of-School Time Programs Collect and Use Data

High-quality out-of-school-time programs are looking for ways to get a better handle on their impact in areas including equity and long-term outcomes.

February 2024

Download Report

Courageous sailing program. Children working on a project.

Table of Contents

Overview

Research Approach

Implementation Tips

Ideas In Practice

Author(s)
Hannah Lantos, Zakia Redd, Jessica Warren, Michael Bradley, and Sham Habteselasse

Publisher(s)
Child Trends

Page Count 62 pages

Summary

How we did this

The authors reviewed research about measuring the work and impact of afterschool, summer, and other out-of-school-time (OST) programs. They found 283 studies on the subject. In 2021, they then surveyed and interviewed staffers largely from programs nominated by expert practitioners in the field. This provided information from a diverse set of 38 OST providers. They ranged from arts programs to civic engagement and workforce preparation programs.

Assessing an organization’s efforts can be a first step to improving them. At youth programs, good data can mean better decisions on staff training, student activities, and other key matters. This report looks at how afterschool, summer, and other out-of-school-time (OST) programs gauge their work and impact. It also examines the obstacles they face in doing so. And it offers recommendations for how to remove those obstacles. It is based in large part on interviews and surveys of staffers from OST programs that experts in the OST field recommended for study.

What Programs Measured

Programs sought to measure outcomes that reflected their content. That content could be arts, workforce readiness, civic engagement, or other subjects depending on the provider’s focus.

Programs also measured social and emotional outcomes. Outcomes of interest to their funders figured prominently, too, such as youth participation rates and program quality measures.

Equity
Program staffers reported thinking seriously about how to better assess equity in their efforts. They were interested in several things here. One was whether program participants from different backgrounds equally experienced positive outcomes. Another was whether the programming advanced participants’ knowledge and skills in areas like civic engagement and social justice. Many staffers expected to see changes in their programs that reflect heightened concern about equity. Those changes, they said, could help them decide what to measure in the future.

For many programs, data collection can be a source of frustration for staff—either because the data they collect isn’t usable or the logistics of collecting it are burdensome.

How Programs Measured Their Work
Programs generally used surveys, interviews, and other conventional tools to assess their efforts. Some turned to less traditional methods like reviewing participant journals and portfolios. One program tracked the number of award tags it gave to participants who had excelled in areas the program focused on.

Challenges Programs Faced

Staffers reported limitations in measurement. Some said they lacked strong tools to assess social-emotional outcomes. They also would have liked to gauge certain important aspects of their work but were unable to. These included:

Longer-term outcomes, such as participants’ college matriculation rate, employment and earnings, or degree of civic involvement
Young people’s behavior
Academic indicators such as test scores
The relationship between staff improvements and youth outcomes.

Longitudinal data proved tough to collect for several reasons. School districts typically closely guard student data and will not release it without a formal agreement, for example. Tracking participants after they leave a program can be especially challenging.

Staffers reported, too, that data gathering sometimes burdened two key groups. One was time-pressed staff members, especially those with little training in data gathering. The other was program participants weary of all the testing they face in school. Funder reporting requirements could be weighty as well. Adding to the burden was that various funders often asked for the same information but in different formats.

It will be important to revisit the question of how OST programs measure equity … as they begin to settle on and codify their approaches.

Key Takeaways

Programs measured outcomes that reflected their content. They also generally assessed social-and-emotional outcomes and outcomes of interest to funders including program participation rates and quality.
Programs were thinking hard about how to better measure equity in their efforts. Approaches included disaggregating data by race and other variables, and tracking development of equity-related knowledge and skills.
Surveys or other conventional methods of gathering data were common. Some programs turned to less traditional methods. These ranged from reviewing participant portfolios to tracking a program’s awards to participants.
Programs faced several challenges in measuring their work and its impact. One was lack of easily accessible long-term data from school districts. Another was burdensome funder requirements.