Nov 21 -- The Evaluation and Assessment Capability (EAC) Section of the National Science Foundation (NSF) Office of Integrative Activities (OIA) invites public comment by January 22, 2024 regarding the proposed information collection for the Evaluation of the National Science Foundation's (NSF) Broader Impacts Review Criterion.
NSF is conducting an evaluation to assess (1) how NSF's Broader Impacts review criterion is applied across the Foundation and (2) its effectiveness in meeting the goals established in section 526 of the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Reauthorization Act of 2010 (42 U.S.C. 1862p-14) (America COMPETES Reauthorization Act of 2010). This evaluation is congressionally directed in section 10341 of the Creating Helpful Incentives to Produce Semiconductors [CHIPS] for America Fund Act 2022.
As part of the evaluation, NSF is conducting a literature review, document analysis, extant data analysis, interviews with NSF staff, and focus groups with NSF principal investigators (PIs) and reviewers. NSF will map findings from the evaluation activities to current NSF policies and practices to identify strategies for improving how NSF applies the review criterion.
The subject of this request is related to the planned focus groups with PIs and reviewers. The focus groups will answer the following research questions (RQs):
RQ1. In what ways do the interpretations of the Broader Impacts review criterion among PIs and reviewers vary, and what factors might contribute to these variations?
RQ2. How do external reviewers assess the Broader Impact review criterion?
RQ3. In what ways do PIs and reviewers perceive that variations in interpretation and assessment can advance or hinder the merit review of proposals?
Findings from the focus groups described in this request will be used to inform interpretation of other evaluation activities within the larger project (including informing interpretation of interviews with NSF staff, document review analyses, and interpretation of extant data analysis of review analyses). For example, we anticipate that participants in these focus groups may raise issues around their understanding and interpretation of Broader Impacts, which can be compared to perceptions that NSF staff report during interviews.
NSF sets forth an ambitious vision for the United States: a nation that leads the world in science and engineering research and innovation, to the benefit of all, without barriers to participation. Toward this end, NSF promotes the progress of science by investing in research and capacity-building activities that expand knowledge in science, engineering, and education. In fiscal year (FY) 2022, NSF evaluated almost 40,000 proposals for research and education activities, making nearly 11,000 new awards totaling more than $8.5 billion.
At the cornerstone of NSF's mission and its investments is its merit review process. NSF program directors with technical and programmatic expertise lead this process, with support from external experts who help evaluate submitted proposals for two main criteria: (1) Intellectual Merit—the potential to advance knowledge; and (2) Broader Impacts—the potential to contribute to society and achieve specific, desired societal outcomes. With these two criteria, NSF has established a commitment to projects that provide tangible benefits to society beyond advancing knowledge.
It is critically important that NSF implement its merit review process in a way that is fair, thorough, competitive, and transparent, and that those internal and external to NSF recognize the process as such. However, as NSF noted, PIs and reviewers might lack clarity about the Broader Impacts criterion, despite NSF's efforts to provide additional guidance. NSF has also noted a lack of consistency in how NSF implements the criterion across directorates, divisions, and programs. Specific challenges related to the understanding and application of Broader Impacts include a lack of consensus on how to define Broader Impacts, and a disconnect between the Broader Impacts requirements stated in the NSF Proposal & Award Policies & Procedures Guide and how panelists review these activities (National Alliance for Broader Impacts 2018).
The purpose of this work, then, is to “assess how the Broader Impact review criterion is applied across the Foundation and make recommendations for improving the effectiveness for meeting the goals established in section 526 of the America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Reauthorization Act of 2010 (42 U.S.C. 1862p-14)” (America COMPETES Reauthorization Act of 2010). This evaluation is congressionally directed in section 10341 of the Creating Helpful Incentives to Produce Semiconductors [CHIPS] for America Fund Act 2022.
Focus groups will be conducted with two types of respondents: Pls and reviewers.
The evaluation will include three PI focus groups of up to seven people each. Participants in these groups will be PIs who submitted a proposal within the last five years. The study team will select participants via a stratified random sample by NSF directorate, institutional characteristics (such as Carnegie classification, MSI status, and locale), and participant characteristics (such as race/ethnicity, gender, years since terminal degree, and new investigator status). PIs have firsthand experience addressing the Broader Impacts review criterion in their proposals. Among this group, key insights include the following:
1. Questions they have about how to address the Broader Impacts review criterion in their research and proposals.
2. Strategies they have employed as a PI in addressing the Broader Impacts review criterion in their research and proposals.
3. Resources or supports received from their respective institutions for developing well-thought-out proposals that address the Broader Impacts review criterion.
Reviewer focus groups will consist of three focus groups of up to seven people each. Participants in these groups will be people who served on a review panel within the last five years. The study team will select participants via a stratified random sample by directorate and participant characteristics (such as how long they have been reviewing NSF proposals). Reviewers have firsthand knowledge about applying the Broader Impacts review criterion. Among this group, key insights include the following:
1. Interpretating and applying the criterion as a reviewer (and compared with as a PI).
2. Reviewer training and guidance.
Contractor: Mathematica
Performance period: June 26, 2023 - December 25, 2024
Obligation: $1,176,472.00
https://www.fpds.gov/common/jsp/LaunchWebPage.jsp?command=execute&requestid=168738915&version=1.5
42 USC §1862p–14. Broader Impacts Review Criterion [America COMPETES Act of 2010 Sec. 526]
(a) Goals -- The Foundation shall apply a broader impacts review criterion to identify and demonstrate project support of the following goals:
(1) Increasing the economic competitiveness of the United States.
(2) Advancing of the health and welfare of the American public.
(3) Supporting the national defense of the United States.
(4) Enhancing partnerships between academia and industry in the United States.
(5) Developing an American STEM workforce that is globally competitive through improved pre-kindergarten through grade 12 STEM education and teacher development, and improved undergraduate STEM education and instruction.
(6) Improving public scientific literacy and engagement with science and technology in the United States.
(7) Expanding participation of women and individuals from underrepresented groups in STEM.
(b) Policy -- Not later than 6 months after January 4, 2011, the Director shall develop and implement a policy for the Broader Impacts Review Criterion that—
(1) provides for educating professional staff at the Foundation, merit review panels, and applicants for Foundation research grants on the policy developed under this subsection;
(2) clarifies that the activities of grant recipients undertaken to satisfy the Broader Impacts Review Criterion shall— (A) to the extent practicable employ proven strategies and models and draw on existing programs and activities; and (B) when novel approaches are justified, build on the most current research results;
(3) allows for some portion of funds allocated to broader impacts under a research grant to be used for assessment and evaluation of the broader impacts activity;
(4) encourages institutions of higher education and other nonprofit education or research organizations to develop and provide, either as individual institutions or in partnerships thereof, appropriate training and programs to assist Foundation-funded principal investigators at their institutions in achieving the goals of the Broader Impacts Review Criterion as described in subsection (a); and
(5) requires principal investigators applying for Foundation research grants to provide evidence of institutional support for the portion of the investigator’s proposal designed to satisfy the Broader Impacts Review Criterion, including evidence of relevant training, programs, and other institutional resources available to the investigator from either their home institution or organization or another institution or organization with relevant expertise.
42 USC 19051: Broader impacts [CHIPS and Science Act of 2010 Sec. 10341]
(a) Assessment -- Not later than 120 days after August 9, 2022, the Director shall enter into an agreement with a qualified independent organization to assess how the Broader Impacts review criterion is applied across the Foundation and make recommendations for improving the effectiveness for meeting the goals established in section 1862p–14 of this title.
(b) Activities -- The Director shall make awards on a competitive basis, to institutions of higher education or non-profit organizations (or consortia of such institutions or organizations) to support activities to increase the efficiency, effectiveness, and availability of resources for implementing the Broader Impacts review criterion, including-
(1) training and workshops for program officers, merit review panelists, award office administrators, faculty, and students to improve understanding of the goals and the full range of potential broader impacts available to researchers to satisfy this criterion;
(2) repositories and clearinghouses for sharing best practices and facilitating collaboration; and
(3) tools for evaluating and documenting societal impacts of research.
NSF Broader Impacts:
https://new.nsf.gov/funding/learn/broader-impacts
NSF DCL A Broader Impacts Framework for Proposals Submitted to NSF's Social, Behavioral, and Economic Sciences Directorate (2021):
https://www.nsf.gov/pubs/2021/nsf21059/nsf21059.jsp
Evolution of Broader Impacts Report (2018):
https://researchinsociety.org/resource/evolution-of-broader-impacts/
Draft data collection instruments and technical documentation requested from NSF by AEA CGR.
FRN:
https://www.federalregister.gov/d/2023-25718