The evaluation involved a randomized controlled trial. The authors write that randomization is the best procedure for guarding against selection effects in which youth more likely to improve over time are disproportionately located in the treatment group. In that event, it is impossible to sort out whether effects are due to selection (who got into the program) or the program itself (the experiences of youth while in the program). This is why randomized controlled trials are typically considered the “gold standard” in evaluation research. 

The authors assessed 13 apprenticeships and their respective control groups for a total of 535 youth. The selection of apprenticeships focused on those instructors who had a history of implementing the After School Matters (ASM) model well. Thus, they did not study the average ASM apprenticeship, but rather what were identified as the better apprenticeships based on ASM nominations and prior evaluation data. 

ASM had led the authors to believe that few alternative after-school activities were available in these schools and communities. However, they discovered that the overwhelming majority of control youth (91%) were involved either in an organized afterschool activity (primarily) or paid work. This changed their understanding of the experimental contrast in an important way: instead of comparing ASM to no treatment, they were actually comparing ASM to the range of organized after-school activities in which these youth normally participate—in effect, an alternative treatment comparison. 

Apprenticeships were located across 10 Chicago public high schools. Students selected an apprenticeship in which they were interested, and those applicants were randomly assigned to the apprenticeship or to the control condition (“business as usual”) by the research team. The majority of participants were African American (77%) and low income (92% received free or reduced price lunch). 

The researchers collected data on a diverse set of outcome variables representing four broad constructs: positive youth development, marketable job skills, academic performance, and problem behavior. Variables were assessed via multiple sources and methods: surveys of youth, interview ratings by human resource professionals in mock job interviews, archival school records, and observations by the research team. Both intervention and control youth were 3 assessed prior to the onset of the apprenticeship (pretest) and after the end of the spring apprenticeship (posttest). They analyzed the data using a hierarchical linear model (HLM) approach, controlling for key demographic variables and a pretest on the same variable as a covariate (when available).