To understand the relationship between licensing requirements for school principals and the knowledge and skills principals need to promote student learning, the researchers conducted a content analysis of the state administrative code sections that govern principal licensure. Content analysis is a methodology for structuring and analyzing written material. This process enabled the researchers to systematically examine and describe administrator licensing requirements in the states. As such regulations constitute a fluid target, it’s important to note that this report is based on regulatory language available between 2002-2004.

The researchers examined licensure content for all 50 states plus the District of Columbia. In each case, they examined the state’s primary code section regarding principal licensure plus any additional code sections it referenced. Within the code, they examined content that specified a requirement for licensure, such as an academic degree, ethical behavior, or knowledge of curriculum. In fact, these “requirements” constituted their unit of observation (with states being the unit of analysis). Given the way state regulations are written, with separate requirements sometimes contained in a single sentence, the researchers' coded observations often included units as small as sentence clauses. In contrast, they excluded procedural content, such as how to submit a license application to a state, or similar non-germane information.

Next, they created coding categories that identified the conceptual and operational dimensions of state licensing regulations and school leadership knowledge and skills. These categories ranged from the big-picture, policy focuses on individual, organizational, and learning attributes, to specific knowledge and skills. Accordingly, this scheme allowed them to examine four levels of detail, each nested within the one above, for instance: (1) learning focus, (2) academic program requirements, (3) assessment systems, and (4) interpreting state and district learning assessments. In constructing the categories, the researchers identified the individual-focused attributes inductively, whereas they culled the organizational- and learning-focused attributes from management and educational research literature. All the content they examined—each observation—fit into one, and only one, of the coding categories. In addition, the researchers created a coding dictionary that allowed them to code the variety of regulatory language they encountered in a consistent manner. For example, “instruction” coded as a learning focus, but “school” as a type of organization, hence organizational focus. “Partnerships” coded as an external resource, “study of” as coursework, and so on. They also took the regulatory language at face value, without imposing any interpretation of their own.

The coding process itself involved several steps. The researchers both, separately, read through the data, identifying coding units—the requirements; then together compared results and agreed on the final set that they would use in the analysis. Next, to ascertain the reliability of their coding form and to make revisions that would improve it, they piloted the coding process on sample regulatory content. At this and later stages, they separately coded all content, marking up hard-copy texts, compared findings, which surfaced mistakes or disagreements, and finalized results. Agreement, or reliability, between the researchers on the pilot study was 31%. After revisions to the coding form and dictionary, agreement on the raw observations in the study itself improved to 93.27%, and on the corrected observations, 99.85%.

Coding resulted in a set of frequencies (counts) for each coding category. These counts corresponded to the number of times each individual, organizational, or learning attribute appeared in a state’s licensure content. These counts went into an Excel file, which enabled the researchers to summarize the data and to identify patterns in it, such as appeared in the figures and tables in this report regarding policy focus, scope, and depth.