To ascertain the ways that each of the eight districts that it supported had invested in principal preparation, and the relationship between the districts and local universities around the design and delivery of leadership preparation programs, The Wallace Foundation contracted with Education Development Center, Inc. (EDC), to conduct an evaluation. The study was designed to investigate the feasibility and efficacy of the new role of districts in leadership preparation as consumers of preparation program graduates and to assess the resulting impact on districts and leadership candidates. EDC engaged three principal investigators and a team of researchers to develop a case study for each district and conduct a cross-case analysis report on key findings.

The primary data collection methods were a series of in-depth interviews of district and university officials and staff, and program participants; a review of program and district documentation; and administration of an online program features survey.

Sampling

There were three levels of sampling decisions. The first was the selection of eight districts for the case study research. The second sampling decision was the selection of two types of universities in each district: (1) universities that had a working relationship with the local districts for leadership preparation, and (2) at least one local university with a leadership preparation program but no relationship with the district. The third sampling decision pertained to the individuals to be interviewed in each site. The researchers focused primarily on staff who worked directly with the leadership preparation programs (including program directors and faculty), those who made institutional decisions about investments in leadership preparation (including department chairs, deans, human resource directors, and superintendents), and those who might observe the benefits (such as area supervisors or assistant superintendents).

Informed Consent

Before initiating the evaluation research, the institutional review boards of EDC and the universities of the research team members reviewed and approved the research design; interview guides; and informed consent forms for interview, observation, and survey participation for issues pertaining to human subjects and attention to informed consent. In addition, each district and EDC reviewed and signed an MOU about the evaluation research. The MOU stipulated the research purposes, the data collection process, the possible burden on district staff, the plan for both case-specific and cross-case analysis, and the reporting expectations.

In conducting the research, the researchers made clear to the districts, universities, and each interviewee that their participation was voluntary, that no individual would be personally identified, and that the research team would handle all information shared confidentially. Agreement to the taping of all interviews was voluntary as well. Interviewees signed an informed consent form giving permission for the interview, taping, and use of the information for the study’s research purposes.

Data Collection

Using their conceptual design and research team input, the researchers developed a common set of interview protocols to guide the field research. The interview guides were open-ended and designed to investigate each category of the study’s conceptual design as it pertained to the role of each interviewee. All interviews were transcribed and coded using Atlas Ti, following a coding schema based on the interview protocols and the study’s conceptual design.

In addition, the researchers modified the Educational Leadership Preparation Program Features Survey, developed by the University Council for Educational Administration and by the American Educational Research Association’s Learning and Teaching in Educational Leadership Special Interest Group. The survey was fielded online using Survey Monkey.

Data Analysis

The researchers used a three-stage data analysis process for the evaluation research. In the first stage, each research team constructed individual district-focused case studies, using a common case study outline, designed around the conceptual framework and relationships being investigated. Within each case study, researchers explored the features of leadership programs, the differences between the affiliated and non-affiliated universities’ programs, the nature of the interinstitutional relationships, and the effects of the district-supported program on the district and universities. The researchers evaluated the validity of each case in two ways: (1) Each case was reviewed by the three principal investigators, who raised questions about evidence, interpretation, and internal consistency; and (2) each case was submitted to the lead project director in each district for review and feedback on errors in fact, missing information, updates, and differences in interpretation. 

The second stage of the data analysis process was a cross-case analysis, which the research team implemented in two steps. They hosted a two-day analysis meeting of the entire research team to talk through key findings in relationship to their core research questions pertaining to program features and quality, organization of the district-university affiliations, and the relationship between the two. In addition, the research team completed a series of cross-case comparison analytic tables on key study topics focusing on important program features and district and university roles, which became the basis for the final report. 

The third stage was drafting a final report, combining analyses of the eight district case studies, the cross-case analysis table findings, and a review of the research literature. The report was then reviewed by the research team, the advisory committee, the lead project directors in each district, and key Foundation staff for errors in fact, missing information, differences in interpretation, and questions for further investigation. Preliminary research findings were discussed with Foundation staff and selected members of the research community at three stages: in the initial phases of data collection, at mid-course in the data analysis phase, and in the report revision phase.

Research Quality and Integrity

The researchers used multiple strategies to ensure research quality and integrity. They included using protocols of informed consent before interviews and survey participation, as noted above; attending to potential sources of conflict between the research sites and the research team; engaging an advisory committee to provide feedback on all phases of the research study; seeking input from the districts on the findings; and sharing findings with the Foundation throughout the study process.