White Papers, Issue Briefs, and Articles

Our white papers, issue briefs, and articles spotlight areas of improvement for social policies and programs, changing the way we see the world a little bit at a time.

Family Support

Methods for Promoting Open Science in Social Policy Research

This brief, authored by Rachel Holzwart and Hilary Wagner, summarizes key themes from the 2019 Office of Planning, Research, and Evaluation Methods Meeting on promoting open science in social policy research. “Open science” is a broad movement to make all phases of research—from design to dissemination—more transparent, accessible, and replicable. Open science methods have gained momentum as highly publicized news stories related to data manipulation (e.g., p-hacking), publication bias (e.g., no publication of null results), and inability to replicate or reproduce research results have cast doubt on research credibility. Proponents of open science strive to transform the research culture using a range of methods, such as preregistering evaluation plans and providing open access to code and data, to encourage open sharing of research information and enable researchers to verify and build on one another’s work.

Download

How Even the Best Evidence Can Yield Bad Decisions, and What We Can Do About It

While evaluation evidence is the best evidence for identifying “what works,” it is still imperfect. In evidence-based policymaking, the imperfections create a significant risk we will implement programs that do not work, and we will suffocate programs that do—or at least can—work. Numerous factors, including limitations of external validity, an overreliance on p-values and hypothesis testing, underpowered research, and even our intolerance for false positives can lead to incorrect conclusions. This paper discusses various ways evidence-based decisions can still be bad decisions. The paper identifies two trends that can address these shortcomings: the use of Bayesian statistical methods and continuous quality improvement. The paper concludes with six recommendations for strengthening evidence-based policymaking.

Download

Understanding Rapid Learning Methods: Frequently Asked Questions and Recommended Resources

This brief, authored by Rachel Holzwart, Robbie Skinner, and Debra Wright, highlights a list of frequently asked questions about rapid learning methods. The list includes links to resources for further information on topics such as the purpose of rapid learning methods, which rapid learning methods are appropriate for which contexts, and practical resources to help programs implement these methods. If you’ve been wanting to learn more about rapid learning methods and their application in social service settings, this is a great place to start.

Download

Rapid Learning: Methods for Testing and Evaluating Change in Social Service Programs

This brief, authored by Rachel Holzwart and Hilary Wagner, summarizes the 2018 Office of Planning, Research, and Evaluation (OPRE) Innovative Methods Meeting discussions on rapid learning methods (e.g., rapid cycle evaluation and continuous quality improvement). It describes considerations when selecting these methods, how researchers have successfully used them, and how to build a lasting culture of learning within an organization.

Download

Rapid Learning: Methods to Examine and Improve Social Programs

This brief, authored by Insight’s vice president Scott Cody and colleague MaryCatherine Arbour of Harvard and Brigham and Women’s Hospital, is based on a presentation delivered during the 2018 Office of Planning, Research, and Evaluation (OPRE) Innovative Methods Meeting on the topic of rapid learning methods.  It includes (1) a definition of rapid learning methods, (2) a guiding framework of questions to design an optimal rapid learning approach, and (3) suggested steps federal agencies can take to promote the effective use of rapid learning methods. To view a video from their presentation and read the published issue brief on the topic, check out https://www.acf.hhs.gov/opre/resource/rapid-learning-methods-to-examine-and-improve-social-programs?fbclid=IwAR3tkaA1LMRdENmAlYjm43NG4LcQJxewkH6fMJCyr95q8yV_VzMlI8ylBDI A version of this video with audio description can be accessed at https://youtu.be/Iaj4scWRDE8.

Download

Understanding Bayesian Statistics: Frequently Asked Questions and Recommended Resources

Bayesian methods are emerging as the primary alternative to p-values, with advantages over the traditional frequentist framework. This guide is intended for readers who wish to better understand or employ Bayesian statistical approaches. Authored by Rachel Holzwart, Hilary Sama, and Debra Wright, it provides short answers to four common questions about Bayesian methods and provides a curated list of resources (including journal articles, book chapters, online courses, and blogs) for further reading.

Download

Bayesian Methods for Social Policy Research and Evaluation

OPRE’s 2017 Methods Meeting on Bayesian methods encouraged attendees to think critically about commonly used statistical methods and approaches. The meeting introduced Bayesian methods as an alternative to support evaluation design and analysis and decisionmaking. This brief, authored by Rachel Holzwart and Debra Wright, summarizes key discussion topics from that meeting. It describes the advantages and disadvantages of Bayesian methods, Bayesian model building and evaluation, examples of Bayesian applications in the field, and how social policy researchers and stakeholders can use Bayesian methods to better support decisionmaking.

Download