OST Outcomes Measurement Summary
Use this resource to review information on tools that can be used by OST practitioners and researchers to assess outcomes for program participants.
Measuring Youth Development
How Out-of-School Time Programs Collect and Use Data
Use this resource to review information on tools that can be used by OST practitioners and researchers to assess outcomes for program participants.
February 2024
Table of Contents
Document
- Author(s)
- Hannah Lantos, Zakia Redd, Jessica Warren, Michael Bradley, and Sham Habteselasse
- Publisher(s)
- Child Trends
Page Count
62 pages
- DOI Link
- https://doi.org/10.56417/3703u3452e
Summary
How we did this
The authors reviewed research about measuring the work and impact of afterschool, summer, and other out-of-school-time (OST) programs. They found 283 studies on the subject. In 2021, they then surveyed and interviewed staffers largely from programs nominated by expert practitioners in the field. This provided information from a diverse set of 38 OST providers. They ranged from arts programs to civic engagement and workforce preparation programs.
Assessing an organization’s efforts can be a first step to improving them. At youth programs, good data can mean better decisions on staff training, student activities, and other key matters. This report looks at how afterschool, summer, and other out-of-school-time (OST) programs gauge their work and impact. It also examines the obstacles they face in doing so. And it offers recommendations for how to remove those obstacles. It is based in large part on interviews and surveys of staffers from OST programs that experts in the OST field recommended for study.
What Programs Measured
Programs sought to measure outcomes that reflected their content. That content could be arts, workforce readiness, civic engagement, or other subjects depending on the provider’s focus.
Programs also measured social-emotional outcomes. Outcomes of interest to their funders figured prominently, too, such as youth participation rates and program quality measures.
Equity
Program staffers reported thinking seriously about how to better assess equity in their efforts. They were interested in several things here. One was whether program participants from different backgrounds equally experienced positive outcomes. Another was whether the programming advanced participants’ knowledge and skills in areas like civic engagement and social justice. Many staffers expected to see changes in their programs that reflect heightened concern about equity. Those changes, they said, could help them decide what to measure in the future.
For many programs, data collection can be a source of frustration for staff—either because the data they collect isn’t usable or the logistics of collecting it are burdensome.
How Programs Measured Their Work
Programs generally used surveys, interviews, and other conventional tools to assess their efforts. Some turned to less traditional methods like reviewing participant journals and portfolios. One program tracked the number of award tags it gave to participants who had excelled in areas the program focused on.
Challenges Programs Faced
Staffers reported limitations in measurement. Some said they lacked strong tools to assess social-emotional outcomes. They also would have liked to gauge certain important aspects of their work but were unable to. These included:
- Longer-term outcomes, such as participants’ college matriculation rate, employment and earnings, or degree of civic involvement
- Young people’s behavior
- Academic indicators such as test scores
- The relationship between staff improvements and youth outcomes.
Longitudinal data proved tough to collect for several reasons. School districts typically closely guard student data and will not release it without a formal agreement, for example. Tracking participants after they leave a program can be especially challenging.
Staffers reported, too, that data gathering sometimes burdened two key groups. One was time-pressed staff members, especially those with little training in data gathering. The other was program participants weary of all the testing they face in school. Funder reporting requirements could be weighty as well. Adding to the burden was that various funders often asked for the same information but in different formats.
It will be important to revisit the question of how OST programs measure equity … as they begin to settle on and codify their approaches.
Key Takeaways
- Programs measured outcomes that reflected their content. They also generally assessed social-emotional outcomes and outcomes of interest to funders including program participation rates and quality.
- Programs were thinking hard about how to better measure equity in their efforts. Approaches included disaggregating data by race and other variables, and tracking development of equity-related knowledge and skills.
- Surveys or other conventional methods of gathering data were common. Some programs turned to less traditional methods. These ranged from reviewing participant portfolios to tracking a program’s awards to participants.
- Programs faced several challenges in measuring their work and its impact. One was lack of easily accessible long-term data from school districts. Another was burdensome funder requirements.
Materials & Downloads
Topics:
Now Viewing:
Overview
Next: Research Approach
