It can be difficult for reporters who write about schools to know what to think about education research. Not all studies are created equal—so how can busy journalists make the best use of their time when considering whether to cover one?
This problem isn’t academic for The Wallace Foundation, given that disseminating key research findings from the work of our grantees is central to the foundation’s mission. That’s why Edward Pauly, our director of research, was happy to provide some practical suggestions at the Education Writers Association’s National Seminar this year. He also used the opportunity to explain a little about how and why the foundation conducts research.
Pauly was joined on the panel by Denise-Marie Ordway, an award-winning reporter who runs the Journalist’s Resource project at Harvard University’s Shorenstein Center on Media, Politics, and Public Policy. She is also on the EWA board of directors.
While the audience was journalists and communications professionals, the experts’ advice also holds true for many other users of education research, including nonprofit and advocacy organizations that use research of any kind. Further, the advice offered by Ordway and Pauly can be applied to studies on topics other than education.
“If a study seems too good to be true, it probably is,” Pauly warned. “Most research findings are not that surprising: They have ‘face validity.’ If you’re startled and shocked, that’s probably a good reason to be skeptical and be careful.”
Ordway cautioned reporters not to assume that a research study is high quality based on the institutions of the people involved: “It isn’t bullet proof just because it’s from Harvard or Stanford.”
Among other helpful points, she steered reporters away from spending too much time on the study’s “abstract,” which summarizes its conclusions. What researchers consider the important takeaways don’t always match what might interest a reporter, she said, explaining that “golden nuggets” of interesting facts and data points are often found deeper into a report.
Pauly summarized his recommendations into six tips for writing about research:
- Look for “literature reviews” of all high-quality research on a topic. Peer-reviewed journals such as Review of Educational Research synthesize the best studies on a topic. Evidence from many studies is more meaningful than evidence from a lone study.
- Find an unbiased, “in-the-know” academic source to share the study with and ask what seems important, reliable, special and valuable about it. What does it add to what we already knew—and why should we believe it?
- Spend more time on studies that are reliable and broad: major, multi-site studies rather than single-site studies, and studies with a “control” group that allows comparisons to be made and differences to be attributed to the program or intervention being evaluated.
- Check out the ranking of a journal to determine its reliability. For example, many journals have Wikipedia articles that provide these rankings.
- Make sure that the study considers alternative explanations for its findings and is clear about its limitations.
- Consider the type of study—student outcomes, implementation of a program or initiative, opinion survey—and evaluate whether the claims it makes are consistent with that kind of study.
In determining what research would be relevant to their audiences, education journalists can ask their sources, “What is it you don’t know that, if you knew it, it would enable you to make a breakthrough in your work?” and then track down the best studies on those important topics, Pauly said.
And, he noted, that’s exactly how The Wallace Foundation decides how to make its grants, seeking to produce answers to big questions that would benefit the field.
You can watch the full EWA session on this recorded Facebook Live session and see the PowerPoint presentation below.
