Scientific Peer Review During the Pandemic and Beyond: Changes, Benefits, and ImpactsPosted on by
The COVID-19 pandemic began significantly impacting the operations of the National Institute for Occupational Safety and Health (NIOSH) in March 2020. Travel restrictions were implemented, and CDC employees ordered to work from home. The Office of Extramural Programs (OEP) Review Team had a confirmed in-person peer review meeting for World Trade Center Health Program (WTCHP) research applications occurring within a week. Suddenly it was impossible to convene this, or any, in-person panel to review applications.
NIOSH/OEP adapted quickly to move from in-person peer review meetings to online review meetings. NIOSH was able to conduct the WTCHP meeting using the Virtual Meeting (VM) module within the NIH Internet Assisted Review (IAR) system, which was already in use for data collection in face-to-face meetings. Thus, no additional infrastructure was required. The IAR system was selected over teleconference because of the large size of the meeting and limited access to teleconference lines.
Panelists were notified of the change and travel, hotel, and conference space were cancelled. The 32 reviewers were immediately given orientation to the IAR Virtual Meeting system and the original meeting dates remained unchanged. The meeting successfully provided a high-quality review of the 35 applications.
Since March 2020, all 42 additional review panel meetings have been virtual, either online (VM or Zoom) or telephone. In FY2022, all 15 review panels were held using Zoom.
Benefits and Challenges
Benefits associated with this change include reduced time burden on reviewers and significantly lower meeting costs due to the absence of travel, meeting space, and conference services. Honorariums for participation continue to be paid to panelists. New meeting formats and procedures have increased the efficiency of the review process.
There are some challenges, however, in managing the large number of reviewers needed with large numbers of applications, especially if they are complex and multi-component. To address the challenge of managing these large meetings virtually, OEP initiated the use of multiple panels to complete high quality reviews in a timely manner and reduce peer reviewer burden.
Minor drawbacks discovered with using virtual meetings include managing quorum and providing opportunities for reviewers to interact before and after meetings. With reviewers from different time zones, ensuring that a minimum number of reviewers are present to achieve quorum can be challenging. If reviewers turn off their video when taking breaks, the Scientific Review Officer assigned to the meeting will know the number of reviewers present at the meeting at any given time.
Reviewers see networking opportunities associated with face-to-face meetings as a benefit of participating in reviews. While some have expressed concern about the lack of networking opportunities with virtual reviews, it has not impacted recruiting, most likely because traveling is not required. While not a replacement for face-to-face networking, NIOSH has allowed time after the meetings for virtual discussion among participants.
As we continue using virtual meetings, we will refine our approaches and adopt new tools and technologies. Upon return to in-person meetings, likely on a limited basis, lessons learned, such as the use of panels to manage large meetings, will be applied. A paradigm shift has occurred, is now accepted, and will continue to develop.
Please share the benefits or challenges you have experienced associated with the use of virtual meetings in the comment section below.
Michael Goldcamp, PhD, is the Review Team Lead in the NIOSH Office of Extramural Programs.
Melinda Sinkule, MA, is a Grants Technical Assistant with the NIOSH Office of Extramural Programs.
JoAnne Fairbanks is a Committee Management Specialist with the NIOSH Office of Extramural Programs.
W. Allen Robison, PhD is the Director of the NIOSH Office of Extramural Programs.