The American Institute of Biological Sciences (AIBS) has published findings from research it conducted on the relationship between panel discussion and scoring in teleconference and face-to-face scientific peer-review panels. The study, part of AIBS' Science of Peer Review initiative, appears in the journal BMJ Open.
Peer-review is a process used by most governmental and many non-governmental scientific research programs in the United States to review, analyze, and identify the most promising research. Most experts agree that peer-review has played a critical role in driving the nation's R&D system, which has been the foundation for economic growth and advancements in security, health, and environmental stewardship throughout the 20th and 21st centuries. In 2014 alone, the U.S. government spent approximately $136 billion on R&D.
The study was conducted by AIBS to provide a better understanding of how the use of teleconferencing for grant peer-review panels may alter outcomes as compared to traditional face-to-face peer-review panel meetings.
"A better understanding of how setting or the use of technology influences the dynamics of a peer-review panel will enable the research evaluation community to make more informed funding decisions," said Scott Glisson, AIBS Co-Interim Executive Director. "To fund the highest impact research we need to understand how factors that can influence the functioning of a peer-review panel may impact deliberations and ultimately funding decisions."
This study considered the influences of the use of teleconferencing because funding agencies are often interested in using this technology to reduce costs and the time burden placed on panel members who travel to on-site panel meetings.
Previous research by AIBS found that discussion times were significantly shorter for teleconference settings as compared to face-to-face, but the influence of discussion on application scoring was unclear. This analysis compared proposal scores by reviewers before and after the peer-review meeting. The authors measured the magnitude and direction of score changes. Comparisons of these score shifts were made for face-to-face and teleconference settings, which provided insights into the effect of communication medium on the subsequent scoring patterns.
"Scoring shifts post-discussion were, on average, small in both settings. Discussion was important for at least 10% of applications, regardless of setting, with these applications moving over the threshold to receive funding or not," said Dr. Stephen Gallo, an author of the study. "Small, but statistically significant differences in post-discussion scoring patterns were uncovered between settings, including a decrease in the magnitude of score shifts in the teleconference panels as compared to face-to-face. However, discussion time had little influence on the magnitude of these score shifts."
Interestingly, panel discussion was found to often result in poorer scores when compared to the initial premeeting scores. In other words, review scores worsened after panels came together and discussed a proposal. This was true regardless of setting.
The subtle differences observed between settings were potentially due to reduced reviewer engagement in teleconferences. More research is needed to understand the extent of this phenomenon, as well as the psychology of decision-making, team performance, and persuasion to better elucidate the effects of peer-review panel setting.
Feeling lonely? It may affect how your brain reacts to food, new research suggests