National Science Board Requests Comments on Draft NSF Merit Review Criteria

Share This

Share/Save
Publication date: 
29 June 2011
Number: 
78

More  than 5,100 individuals responded to a National Science Board task force when it  requested input on the criteria that is used to evaluate proposals submitted to  the National Science Foundation.  Now the  task force is asking the science community to review proposed clarifications to  the two criteria.  Comments are to be  submitted by July 14.

The  foundation receives approximately 45,000 proposals every year.  About 11,500 of these proposals receive new  funding awards.  Future awards will be  determined using the new criteria if they are adopted. 

The  twenty-five members of the National Science Board establish the  policies of the NSF.  In 1997 the two  part Merit Review Criteria consisting of Intellectual Merit and Broader Impacts  was implemented following a review by the Board and foundation staff.  At the request of Congress, the Board  assessed these criteria in 2005, issuing a report stating that the foundation’s  merit review process “remains an international “gold standard’ for review of science  and engineering research proposals.” 

In  response to concern that has been expressed about the application of this  criteria the Board established an eleven member Task Force on Merit Review,   chaired by Alan Leshner.   The charge to  the Task Force, selections of which are below, describes the concern about the  review criteria:

“Five  years have passed since the last review of the Merit Review process and a new  National Science Foundation Strategic Plan will be issued shortly. Moreover,  the current review criteria have now been in effect for over a decade, and in  light of reports of some confusion in the field and inconsistency of their  application and impact, it is timely for the National Science Board both to  evaluate the current criteria with respect to their definitions and the way  they are applied to the NSF portfolio of increasingly complex and  interdisciplinary projects, and to ask whether the Merit Review process could  be enhanced or modified, by clarifying or amending the statements of the Merit  Review Criteria.

“The  NSB Task Force on Merit Review is hereby reconstituted at the February 3-4,  2010 National Science Board meeting. The Task Force is charged with examining  the two Merit Review Criteria and their effectiveness in achieving the goals  for NSF support for science and engineering research and education. This may  include revising the merit review methodology, revising one or both of the  merit review criteria and the way they are interpreted and applied, or the task  force may find that the methodology and criteria are clear and function as  intended with no further changes or action required.”

A  “review-in-brief” of the current two-part criteria highlighted issues that the  task force examined.  Regarding  Intellectual Merit, the brief explains:

“A  critical criterion for NSF’s funding of research has been the proposed  project's intellectual merit, both in overall quality and in significance to  the broader field. A concern has arisen over the past few years, however, that  the current system is missing the importance of some more transformative (often  also called high-risk, high-payoff) research and that the system has become a  bit more conservative as funds have become more constrained, despite efforts by  NSF to emphasize transformative research.”

Regarding  Broader Impacts, the brief states:

“The  Broader Impacts criterion identifies the important outcomes and consequences of  NSF-supported research. Anecdotal evidence suggests that this requirement can  be very confusing to the research community, which continues to express  frustration in interpreting and thus responding effectively to the Broader  Impacts criterion when creating a proposal.”

It  continues:

“In  July 2007, Merit Review Broader Impacts Criterion: Representative Activities (http://www.nsf.gov/pubs/gpg/broaderimpacts.pdf)  was made available to PIs in the Grant Proposal Guide, which includes examples  of ways that broader impacts could be incorporated into research projects.  These examples are quite diverse but that diversity can also make them  confusing to proposers and to NSF program staff attempting to address the  Broader Impacts criterion in the review and decision process. There also is  concern that these examples can appear to be directive yet are not fully  inclusive. For example, they do not fully reflect the importance of impacts on  such issues as innovation, national security and economic growth. Finally,  there appears to be substantial confusion about how best to meet the  requirements of this criterion, whether on an individual project level or at  the proposing institution level.”

In  conducting this year-long review, the task force interacted using various means  with the research communities and institutions, and through a survey of NSF  program officers and staff. Stakeholder  views were solicited through an NSF web site. 

On  June 14, the Board issued “NSB/NF Seeks Input on Proposed Merit Review Criteria  Revision and Principles.”  The Board explained:

“One of the most striking  observations that emerged from the data analyses was the consistency of the  results, regardless of the perspective. All of the stakeholder groups  identified similar issues, and often offered similar suggestions for  improvements. It became clear that the two review criteria of Intellectual  Merit and Broader Impacts are in fact the right criteria for evaluating NSF  proposals, but that revisions are needed to clarify the intent of the criteria,  and to highlight the connection to NSF’s core principles.”

The Board released “two draft  revised criteria and the principles upon which they are based” that total  approximately 430 words.  “It is expected  that NSF will develop specific guidance for PIs, reviewers, and NSF staff on  the use of these criteria after the drafts are finalized. Your comments will  help inform development of that guidance, and other supporting documents such  as FAQs” the notice states.  National  Science Board Chairman Ray Bowen and National Science Foundation Director Subra  Suresh conclude this notice stating “We do hope that you will share your  thoughts with us.” Comments on the draft  Merit Review Principles and Criteria   are due by July 14, and should be sent to meritreview [at] nsf.gov

Explore FYI topics: