Our white papers, issue briefs, and articles spotlight areas of improvement for social policies and programs, changing the way we see the world a little bit at a time.
Our white papers, issue briefs, and articles spotlight areas of improvement for social policies and programs, changing the way we see the world a little bit at a time.
We are excited to highlight a new resource list that includes articles and books for readers wanting to learn more about applying qualitative and mixed method approaches to social policy research and evaluation. The list, authored by Westat Insight’s Hilary Wagner for OPRE’s 2022 Methods Meeting, defines qualitative inquiry and highlights how rigor and relevance can be prioritized, identifies key considerations for designing mixed methods studies, indicates strategies for effectively integrating qualitative and qualitative methods, and shows how qualitative and mixed methods approaches can promote equity and inclusion in research.
Download
This brief, prepared by MEF Associates and Westat Insight researchers Lashawn Richburg-Hayes and Allison Hyra, describes how the Race Equity for Fatherhood, Relationship, and Marriage Programs to Empower Black Families (REFRAME) project describes ways to center Black voices in research on programs serving Black families. It is the first brief to describe the REFRAME project and provides important lessons learned relevant for other research and evaluation efforts looking to center their work within the concerns, values, and desires of their population of interest. Objectives are to promote equity, leverage a mostly Black expert workgroup, and uphold the principles of equity throughout.
Download
This brief, authored by Megan Worden, Hilary Wagner, and Debra Wright summarizes key themes from the 2021 Office of Planning, Research, and Evaluation Methods Meeting on Community-Engaged Research. Not only is engaging communities in research and evaluation critical for implementing effective programs, community members can bring valuable input to program evaluation, improving the rigor and relevance of the research. Engaging communities in research and evaluation—especially when identifying research questions, designing and implementing the study or intervention, recruiting study participants, and interpreting findings—may lead to more relevant research questions, greater recruitment success, increased external validity, greater retention of participants, and enhanced understanding of findings (Viswanathan et. al 2004). In this brief we define community engaged research, discuss the motivation for incorporating this approach, and provide examples, tools, and strategies for implementing community-engaged research.
Download
This brief, authored by Rachel Holzwart, Hilary Wagner, and Megan Worden, summarizes key themes from the 2020 Office of Planning, Research, and Evaluation Methods Meeting on using core components approaches in social services settings. Core components of effective interventions are the “essential functions and principles that define the program and are judged as being necessary to produce outcomes in a typical service setting” (Blase & Fixsen). Core components approaches can complement traditional social policy evidence building, provide information needed to successfully adapt programs across a variety of conditions, and determine which intervention elements are critical to achieving desired results and delivering benefit to participants.
Download
This brief, authored by Insight’s Andrew Breck and Breanna Wakar, discusses the key features of a well-designed and implemented subgroup analysis. Subgroup analysis enables researchers to determine whether policy or program impacts vary across groups. While the brief focuses mainly on a multiple regression framework, it also provides an overview of alternative approaches to conducting subgroup analyses.
Download
This resource list, compiled by Insight’s Dani Hansen and Rachel Holzwart, documents references, tools, and websites for readers wanting to learn more about open science methods. “Open science” represents a broad movement to make research, data, and findings more transparent, accessible, and replicable throughout the research process. This document highlights federal initiatives and policies that support open science, the process of registering study plans in a repository before beginning a research project, reproducibility and replicability, and managing and sharing data.
Download
The brief, written by Insight’s Carole Trippe and Brittany McGill, summarizes the results of one of three components of the National Evaluation of ConnectHome, a collaborative effort of communities, the federal government, and the private sector to bring high-speed internet access to low-income families in HUD-assisted housing. The brief describes the Baseline Access Survey—which assessed the baseline level of internet access among families with children in 22 pilot communities prior to implementing the initiative. Insight examined the level of in-home internet access, the types of internet connection, the number and types of devices used (e.g., phone, laptop, tablet), the reasons for lack of internet access, and awareness of the ConnectHome initiative. Results show only about one-third of households have high-speed internet access at home. Another third are “underconnected,” relying, for example, on a cell phone data plan for in-home internet access, and one-third have no internet access at all at home. Most households without internet access report cost as the primary reason.
Download
This brief, authored by Rachel Holzwart and Hilary Wagner, summarizes key themes from the 2019 Office of Planning, Research, and Evaluation Methods Meeting on promoting open science in social policy research. “Open science” is a broad movement to make all phases of research—from design to dissemination—more transparent, accessible, and replicable. Open science methods have gained momentum as highly publicized news stories related to data manipulation (e.g., p-hacking), publication bias (e.g., no publication of null results), and inability to replicate or reproduce research results have cast doubt on research credibility. Proponents of open science strive to transform the research culture using a range of methods, such as preregistering evaluation plans and providing open access to code and data, to encourage open sharing of research information and enable researchers to verify and build on one another’s work.
Download
While evaluation evidence is the best evidence for identifying “what works,” it is still imperfect. In evidence-based policymaking, the imperfections create a significant risk we will implement programs that do not work, and we will suffocate programs that do—or at least can—work. Numerous factors, including limitations of external validity, an overreliance on p-values and hypothesis testing, underpowered research, and even our intolerance for false positives can lead to incorrect conclusions. This paper discusses various ways evidence-based decisions can still be bad decisions. The paper identifies two trends that can address these shortcomings: the use of Bayesian statistical methods and continuous quality improvement. The paper concludes with six recommendations for strengthening evidence-based policymaking.
Download