White Papers, Issue Briefs, and Articles

Our white papers, issue briefs, and articles spotlight areas of improvement for social policies and programs, changing the way we see the world a little bit at a time.

White Papers, Issue Briefs, and Articles

OPRE 2019 Methods Meeting Resources List

This resource list, compiled by Insight’s Dani Hansen and Rachel Holzwart, documents references, tools, and websites for readers wanting to learn more about open science methods. “Open science” represents a broad movement to make research, data, and findings more transparent, accessible, and replicable throughout the research process. This document highlights federal initiatives and policies that support open science, the process of registering study plans in a repository before beginning a research project, reproducibility and replicability, and managing and sharing data.

Promising Practices Brief: Improving Student Engagement and Attendance During COVID-19 School Closures

Nationwide school closures during the COVID-19 pandemic have created an unprecedented level of activity in the online learning space. The move toward remote learning has created questions about whether and how attendance should be taken and measured in an online environment and the importance of meaningful remote engagement that leads to learning. In this promising practices brief, Insight and AIR share an overview of the research on engagement and attendance in online environments, and the findings from our conversations with five districts across the country about their efforts to support student attendance and engagement. This paper was funded by the US Department of Education’s national technical assistance center for addressing chronic absence. Read the Executive Summary HERE.

Methods for Promoting Open Science in Social Policy Research

This brief, authored by Rachel Holzwart and Hilary Wagner, summarizes key themes from the 2019 Office of Planning, Research, and Evaluation Methods Meeting on promoting open science in social policy research. “Open science” is a broad movement to make all phases of research—from design to dissemination—more transparent, accessible, and replicable. Open science methods have gained momentum as highly publicized news stories related to data manipulation (e.g., p-hacking), publication bias (e.g., no publication of null results), and inability to replicate or reproduce research results have cast doubt on research credibility. Proponents of open science strive to transform the research culture using a range of methods, such as preregistering evaluation plans and providing open access to code and data, to encourage open sharing of research information and enable researchers to verify and build on one another’s work.

Students in Subbaccalaureate Health Sciences Programs: 2015–16

A new National Center for Education Statistics Data Point authored by Shannon Griffin and Albert Y. Liu examines the characteristics of students seeking subbaccalaureate credentials (certificates and associate’s degrees) in health sciences. Among subbaccalaureate occupational fields, health sciences was the most common field of study, enrolling more than one in three occupational education students, followed by business and marketing.

Labor Market Outcomes for High School Career and Technical Education Participants: 2016

This National Center for Education Statistics (NCES) data point, authored by Rachel Holzwart and Albert Liu, examines how the early labor market outcomes of public high school graduates vary by the number of career and technical education (CTE) credits earned during high school. We found the unemployment rate for graduates with 3.00 or more CTE credits is less than their peers who earned fewer CTE credits. The Data Point is based on data from the High School Longitudinal Study of 2009 (HSLS:09), which surveyed a nationally representative sample of grade 9 students in 2009, with follow-up data collections in 2012, 2013, and 2016. To view the full report, please visit https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2020060.

How Even the Best Evidence Can Yield Bad Decisions, and What We Can Do About It

While evaluation evidence is the best evidence for identifying “what works,” it is still imperfect. In evidence-based policymaking, the imperfections create a significant risk we will implement programs that do not work, and we will suffocate programs that do—or at least can—work. Numerous factors, including limitations of external validity, an overreliance on p-values and hypothesis testing, underpowered research, and even our intolerance for false positives can lead to incorrect conclusions. This paper discusses various ways evidence-based decisions can still be bad decisions. The paper identifies two trends that can address these shortcomings: the use of Bayesian statistical methods and continuous quality improvement. The paper concludes with six recommendations for strengthening evidence-based policymaking.

Understanding Rapid Learning Methods: Frequently Asked Questions and Recommended Resources

This brief, authored by Rachel Holzwart, Robbie Skinner, and Debra Wright, highlights a list of frequently asked questions about rapid learning methods. The list includes links to resources for further information on topics such as the purpose of rapid learning methods, which rapid learning methods are appropriate for which contexts, and practical resources to help programs implement these methods. If you’ve been wanting to learn more about rapid learning methods and their application in social service settings, this is a great place to start.

Rapid Learning: Methods for Testing and Evaluating Change in Social Service Programs

This brief, authored by Rachel Holzwart and Hilary Wagner, summarizes the 2018 Office of Planning, Research, and Evaluation (OPRE) Innovative Methods Meeting discussions on rapid learning methods (e.g., rapid cycle evaluation and continuous quality improvement). It describes considerations when selecting these methods, how researchers have successfully used them, and how to build a lasting culture of learning within an organization.

Rapid Learning: Methods to Examine and Improve Social Programs

This brief, authored by Insight’s vice president Scott Cody and colleague MaryCatherine Arbour of Harvard and Brigham and Women’s Hospital, is based on a presentation delivered during the 2018 Office of Planning, Research, and Evaluation (OPRE) Innovative Methods Meeting on the topic of rapid learning methods.  It includes (1) a definition of rapid learning methods, (2) a guiding framework of questions to design an optimal rapid learning approach, and (3) suggested steps federal agencies can take to promote the effective use of rapid learning methods. To view a video from their presentation and read the published issue brief on the topic, check out https://www.acf.hhs.gov/opre/resource/rapid-learning-methods-to-examine-and-improve-social-programs?fbclid=IwAR3tkaA1LMRdENmAlYjm43NG4LcQJxewkH6fMJCyr95q8yV_VzMlI8ylBDI A version of this video with audio description can be accessed at https://youtu.be/Iaj4scWRDE8.

Training Professionals in Service Delivery: Key Findings From an Evaluation of the Autism CARES Developmental-Behavioral Pediatrics Training Program

This brief presents findings from a study of the DBP training program grantees awarded through HRSA’s Maternal and Child Health Bureau (MCHB) Autism CARES Act funds. HRSA’s Autism CARES investments have one overarching goal: enable all infants, children, and adolescents who have or are at risk of developing ASD/DDs to reach their full potential by developing a system of services that includes developmental screening of children as early as possible for ASD/DDs, conducting early interdisciplinary evaluations to confirm or rule out ASD/DDs, and providing early evidence-based interventions when a diagnosis is confirmed. The study was part of a comprehensive evaluation designed to assess grantee’s effectiveness in reaching this goal by improving autism spectrum disorder service delivery nationwide. Overall, the evaluation highlighted the efforts of nearly 100 grantees who worked to conduct research, provide professional training, and improve comprehensive coordinated State systems of care for people with autism spectrum disorder.