Systems that coordinate afterschool, summer and other out-of-school-time programming community-wide have emerged in a number of U.S. cities and counties over the last 15 years or so. The organizations that oversee these systems increasingly recognize the need for periodic pulse checks to evaluate their efforts and inform improvements. But what, exactly, should these organizations assess and how?
In 2014, a framework to help answer that was developed by Every Hour Counts, a national coalition of citywide organizations that seeks to increase access to high-quality learning opportunities, particularly for students from underserved communities. This framework is a research-informed update of the tool.
The heart of the framework is 11 desired outcomes of system work, some or all of which system leaders might want to measure progress toward, depending on local needs and circumstances. Five are directly related to overall system work and include whether a common goal for afterschool has been established. Three regard the efforts of programs, stressing, for instance, that they use management practices that enhance program quality. And three are related to young people—the rate of youth participation in programs, among them.
For each of the 11 items, the tool describes indicators signaling progress toward the outcome; the type of data that can be collected for the indicators; ideas for working with the data; and ways to interpret and use the findings. A feature of the update from the 2014 version of the framework is a set of racial equity questions for each outcome, exploring matters ranging from whether system decision-making is inclusive to whether programs distribute high-quality offerings equitably.
Also new is a guide to the tool’s use. Putting Data to Work for Young People: A Guidebook for the Every Hour Counts Framework for Measurement, Continuous Improvement, and Equitable Systems was written by two researchers, Jennifer Sloan McCombs and Anamarie A. Whitaker, who led a RAND Corp. evaluation of how the 2014 version of the framework was used by three different intermediaries—one each in Boston (Boston After School & Beyond); Providence, R.I. (Providence After School Alliance); and St. Paul (Sprockets). That research informed the 2021 framework. The framework and guidebook were developed with support from The Wallace Foundation and The Charles Stewart Mott Foundation.
In all cases, the authors urge care with data gathering and analysis. “To get the benefits of data, afterschool intermediaries need to be thoughtful and deliberate about why they should collect data, what to collect, how to collect it, and how to use it,” they say.
A separate 2019 RAND publication, Putting Data to Work for Young People: A Ten-Step Guide for Expanded Learning Intermediaries, offers insights into gathering, analyzing and managing data.