Projects

Merit Review: Assessment of Investigator and Reviewer Experiences

Project Overview

Insight is leading an evaluation of the National Science Foundation’s (NSF) Merit Review System. The goal of the study is to assess principal investigators and reviewers’ experiences with NSF’s merit review system and to determine the impact of 5 pilot programs aimed at increasing the quality and efficiency of the review system. Currently, proposals submitted to NSF are evaluated against 2 main criteria: intellectual merit and broader impacts. The current challenge facing NSF is the imbalance between the number of proposals received and the number of NSF staff and external reviewers needed to fully support the review process. To address this challenge, NSF conducted 5 pilot reviews between 2012 and 2014 that were designed to ease reviewer burden and increase flexibility for reviewers.

Insight has planned a 2-phase mixed-method research design that aims to 1) assess the current merit review system and 2) evaluate the merit review pilots. As part of this process, Insight will develop and administer a Web survey to about 50,000 principal investigators who submitted and/or reviewed proposals to/for NSF during 2012–2014, conduct in-depth interviews with NSF staff, and examine the impact of the 5 pilot studies through a quasi-experimental design using extant time-series data.

This study will:

  • Investigate the trends in proposal submissions, ratings, and funding over time.
  • Assess the principal investigator and reviewer perceptions of the merit review system.
  • Examine the relationship between the merit review process and principal investigator decisions to submit proposals.
  • Document the implementation of the 5 pilot programs.
  • Determine the impact the pilot studies have had on reviewer workload.

Based on the results of the study, Insight will develop a final report summarizing the findings from the evaluation along with recommendations regarding the implementation of the new pilot programs. The results will be used by NSF to improve the merit review system and inform the further implementation of the current or new pilot studies.

Core Activities

Program Evaluation; Survey Design and Implementation; Survey Research; Data Collection; Qualitative Research; Data Analysis; Report Development & Presentation; Developing/Synthesizing Recommendations

Products

The final report will be submitted in September 2016.