White Papers, Issue Briefs, and Articles

Our white papers, issue briefs, and articles spotlight areas of improvement for social policies and programs, changing the way we see the world a little bit at a time.

Military and Veteran Support

How Even the Best Evidence Can Yield Bad Decisions, and What We Can Do About It

While evaluation evidence is the best evidence for identifying “what works,” it is still imperfect. In evidence-based policymaking, the imperfections create a significant risk we will implement programs that do not work, and we will suffocate programs that do—or at least can—work. Numerous factors, including limitations of external validity, an overreliance on p-values and hypothesis testing, underpowered research, and even our intolerance for false positives can lead to incorrect conclusions. This paper discusses various ways evidence-based decisions can still be bad decisions. The paper identifies two trends that can address these shortcomings: the use of Bayesian statistical methods and continuous quality improvement. The paper concludes with six recommendations for strengthening evidence-based policymaking.

Download