Projects

Merit Review: Assessment of Investigator and Reviewer Experiences

Project Overview

Insight is leading an evaluation of the National Science Foundation’s (NSF) Merit Review Process. Principal investigators (PIs) submit approximately 50,000 proposals to NSF each year, and reviewers evaluate each one against two main criteria: intellectual merit and broader impacts. The goal of this study is to conduct a biennial web survey of PIs’ and reviewers’ experiences with NSF’s review process, with particular focus on burden, satisfaction, and quality. Each survey also includes an examination of new pilot programs aimed at increasing the quality and efficiency of the review system that year.

Survey results have indicated one important component of the merit review process is the quality of written reviews. Recently, NSF set an annual goal to improve the quality of written reviews of NSF proposals but has no way to assess the quality of written reviews. As a secondary component of this study, Insight is—

  • Developing, validating, applying, and automating a tool to assess review quality
  • Using the new review quality tool to assess the impact of several previous pilot approaches on review quality using difference-in-differences analyses
  • Testing the effect of several new, incremental changes to the merit review process using a Plan-Do-Study-Act approach
  • Conducting value stream mapping to identify possible strategies for streamlining the merit review process to reduce dwell time

Products

Final reports, Merit Review—Assessment of Investigator and Reviewer Experiences (2017); Merit Review—2017 Assessment of Investigator and Reviewer Experiences (2018); Merit Review—2019 Assessment of Investigator and Reviewer Experiences (anticipated 2020)