More than 5,100 individuals responded to a National Science Board task force when it requested input on the criteria that is used to evaluate proposals submitted to the National Science Foundation. Now the task force is asking the science community to review proposed clarifications to the two criteria. Comments are to be submitted by July 14.
The foundation receives approximately 45,000 proposals every year. About 11,500 of these proposals receive new funding awards. Future awards will be determined using the new criteria if they are adopted.
The twenty-five members of the National Science Board establish the policies of the NSF. In 1997 the two part Merit Review Criteria consisting of Intellectual Merit and Broader Impacts was implemented following a review by the Board and foundation staff. At the request of Congress, the Board assessed these criteria in 2005, issuing a report stating that the foundation’s merit review process “remains an international “gold standard’ for review of science and engineering research proposals.”
In response to concern that has been expressed about the application of this criteria the Board established an eleven member Task Force on Merit Review, chaired by Alan Leshner. The charge to the Task Force, selections of which are below, describes the concern about the review criteria:
“Five years have passed since the last review of the Merit Review process and a new National Science Foundation Strategic Plan will be issued shortly. Moreover, the current review criteria have now been in effect for over a decade, and in light of reports of some confusion in the field and inconsistency of their application and impact, it is timely for the National Science Board both to evaluate the current criteria with respect to their definitions and the way they are applied to the NSF portfolio of increasingly complex and interdisciplinary projects, and to ask whether the Merit Review process could be enhanced or modified, by clarifying or amending the statements of the Merit Review Criteria.
“The NSB Task Force on Merit Review is hereby reconstituted at the February 3-4, 2010 National Science Board meeting. The Task Force is charged with examining the two Merit Review Criteria and their effectiveness in achieving the goals for NSF support for science and engineering research and education. This may include revising the merit review methodology, revising one or both of the merit review criteria and the way they are interpreted and applied, or the task force may find that the methodology and criteria are clear and function as intended with no further changes or action required.”
A “review-in-brief” of the current two-part criteria highlighted issues that the task force examined. Regarding Intellectual Merit, the brief explains:
“A critical criterion for NSF’s funding of research has been the proposed project's intellectual merit, both in overall quality and in significance to the broader field. A concern has arisen over the past few years, however, that the current system is missing the importance of some more transformative (often also called high-risk, high-payoff) research and that the system has become a bit more conservative as funds have become more constrained, despite efforts by NSF to emphasize transformative research.”
Regarding Broader Impacts, the brief states:
“The Broader Impacts criterion identifies the important outcomes and consequences of NSF-supported research. Anecdotal evidence suggests that this requirement can be very confusing to the research community, which continues to express frustration in interpreting and thus responding effectively to the Broader Impacts criterion when creating a proposal.”
“In July 2007, Merit Review Broader Impacts Criterion: Representative Activities (http://www.nsf.gov/pubs/gpg/broaderimpacts.pdf) was made available to PIs in the Grant Proposal Guide, which includes examples of ways that broader impacts could be incorporated into research projects. These examples are quite diverse but that diversity can also make them confusing to proposers and to NSF program staff attempting to address the Broader Impacts criterion in the review and decision process. There also is concern that these examples can appear to be directive yet are not fully inclusive. For example, they do not fully reflect the importance of impacts on such issues as innovation, national security and economic growth. Finally, there appears to be substantial confusion about how best to meet the requirements of this criterion, whether on an individual project level or at the proposing institution level.”
In conducting this year-long review, the task force interacted using various means with the research communities and institutions, and through a survey of NSF program officers and staff. Stakeholder views were solicited through an NSF web site.
On June 14, the Board issued “NSB/NF Seeks Input on Proposed Merit Review Criteria Revision and Principles.” The Board explained:
“One of the most striking observations that emerged from the data analyses was the consistency of the results, regardless of the perspective. All of the stakeholder groups identified similar issues, and often offered similar suggestions for improvements. It became clear that the two review criteria of Intellectual Merit and Broader Impacts are in fact the right criteria for evaluating NSF proposals, but that revisions are needed to clarify the intent of the criteria, and to highlight the connection to NSF’s core principles.”
The Board released “two draft revised criteria and the principles upon which they are based” that total approximately 430 words. “It is expected that NSF will develop specific guidance for PIs, reviewers, and NSF staff on the use of these criteria after the drafts are finalized. Your comments will help inform development of that guidance, and other supporting documents such as FAQs” the notice states. National Science Board Chairman Ray Bowen and National Science Foundation Director Subra Suresh conclude this notice stating “We do hope that you will share your thoughts with us.” Comments on the draft Merit Review Principles and Criteria are due by July 14, and should be sent to meritreview [at] nsf.gov