How Communities Can Put Data to Work for Young People

​​“Show me the numbers.” It’s a refrain that’s sure to be familiar to those who work hard to create enriching afterschool and summer experiences for young people. Funders and civic leaders want data demonstrating how their dollars are making a difference. Program providers want to use data to get better and make a case for public support. Often, they rely on intermediaries—nonprofit organizations that coordinate out-of-school-time (OST) efforts and resources in a community—to oversee the data gathering and analysis. But what if intermediaries are gathering the wrong data in the wrong way?

As part of a project spearheaded by Every Hour Counts, a coalition representing intermediaries, the RAND Corporation explored how three of the nation’s most mature intermediaries gather and use data. RAND reviewed the quality of the data the organizations collected, the measurement tools and databases they used and more. The result was the recently published Putting Data to Work for Young People: A Ten-Step Guide for Expanded Learning Intermediaries. Developed with support from Wallace and the Charles Stewart Mott Foundation, the guide offers practical advice on gathering and working with data to improve decision making.

Every Hour Counts Executive Director Jessica Donner, along with the heads of the three intermediaries that participated in the project—Erik Skold of Sprockets in St. Paul, Minn., Hillary Salmons of the Providence After School Alliance, and Chris Smith of Boston After School & Beyond—provided us with their take on the power of data and what intermediaries and others in the field can do to harness it.

What is the problem this guide is intended to solve?

Donner: Over the course of our project with RAND and the three intermediaries, we learned that data use is messy, much more so than we anticipated. A number of key questions came up, like, What youth outcomes should we measure to show the impact our out-of-school-time programs are having? What’s the best approach for sharing data with providers and schools? How do we act on data in a timely and meaningful way? We developed the guide to answer those questions and to help intermediaries and others deal with common data-related challenges.

Why is it important for intermediaries to be able to work with data effectively? What does effective data work look like?

Skold: Data is a space where intermediaries can really add value to the field. Youth-serving organizations often lack the time and capacity to effectively collect, analyze and use data. Intermediaries can provide them with the systems, tools and processes to help them better understand what’s happening in their programs and make improvements. Intermediaries can also aggregate data from across programs to tell a broader story about what’s happening in a city or a system. That allows for larger community conversations about how best to support youth and helps policymakers and other stakeholders analyze gaps and needs and think about how to prioritize investments.

Smith: Data is one way for us to know, as a city, if we are effective in our work of providing high-quality opportunities for students to grow in their knowledge and skills. It can shine a light on what we're doing well and what we need to work on, and help programs across the city build capacity together.

How should intermediaries get started working with data? What's the first step?

Donner: There is an overwhelming plethora of data out there for intermediaries to consider so the first step for intermediaries, as noted in the report, is to do some hard thinking about the purpose of the data gathering. Then, we hope communities will turn to the Every Hour Counts Measurement Framework, a tool that our organization has developed (and is revising) to  streamline and simplify a data collection process that can be daunting. The framework lays out what we’ve assessed as the most valuable outcomes for out-of-school-time system builders to measure, how to measure them and the research base for each. The framework has an intentional tri-level focus on a small number of system-, program- and youth-level outcomes that we hope communities will achieve as a result of building local expanded-learning systems. Informed by our work with RAND and our network, we are releasing a revised Framework in 2020 that further distills the outcomes into an even more manageable and focused list for intermediaries.

We advise communities to tread lightly when it comes to measuring youth outcomes. Systems are ultimately developed to support young people, so there’s a natural desire to want to measure the impact of your investment. But positive youth outcomes develop through multiple experiences over the course of a lifetime. We encourage systems to start by focusing on promoting and measuring the conditions that research has shown to improve youth outcomes: program design, high-quality program implementation and frequent attendance.

What are the biggest data-related challenges that OST intermediaries face?

Skold: One of the biggest challenges is working with large amounts of data from various program models. Aggregating and making meaning of data collected from programs of varying focus, length and age groups is difficult. It can be especially difficult when trying to demonstrate the impact that the programs and the intermediary are having on participants. Intermediaries need to be very thoughtful about what data they’re collecting from programs and what types of data to aggregate. 

Donner: Out-of-school-time intermediaries, and they aren’t alone in this, don’t know what they don’t know. Data work is complex, and without tremendous in-house data expertise, it’s hard to know what questions to ask, where to start and where there are missed opportunities for efficiencies. Intermediaries need to develop an infrastructure to do data work well, and that takes support, financial and in-kind, from public and private partners.

What’s one piece of advice you would give to other intermediaries to help them get better at working with data?

Salmons: Take the changes one step at a time. While you may have an ultimate vision to guide you towards where you want to be, understand that each step along the way requires thoughtful planning, trial, reflection and improvement. That way, you can set clear goals of what to accomplish within a set time period; be honest with your partners and stakeholders about what’s realistic within the coming months and years; and make sure that each data improvement you make is meeting their needs.

Smith: As early as you can manage it, set up a system that makes it easier for programs to collect and submit data and designate intermediary staff who can devote time and attention to answering questions, troubleshooting and helping programs see the importance of continuous improvement.

How can intermediaries use the guide to get better at working with data? What are some of the most useful tips that came out of the research?

Donner: One of the most important tips featured in this guide is for intermediaries to find a way to have in an in-house point person for data—even if they work with an outside research firm or university. Due to the complexity of data collection, the intense nature of collaboration with stakeholders and the likelihood of mistakes, we’ve learned it’s a good idea to have a person on the team who’s dedicated to using data effectively and efficiently and, above all, thinking about the right questions to move the work forward.

Another recommendation from the guide is to start by making a list of your key stakeholders, what they need or want to know and how they’re likely to use the information. That advice is everything. We hear time and again that the mayor or superintendent asks a particular question, and intermediaries want to be at the ready with attendance, retention and other stats. So how do you get in front of that, anticipate questions and design a system that gets you the data you need to answer them? The guide has template—data needs for program directors—to help.

Top photo: Youth race cardboard boats they designed and built in one of the many out-of-school time offerings in Providence. Photo courtesy of the Providence After School Alliance.

For more information, see these publications on afterschool and summer programs for a review of evidence about out-of-school-time programming. These Wallace Perspectives offer insights on summer and afterschool as well.