Connect

Peer Review Challenge

October 30, 2012

By Clare Lehane

Winners unveiled

This Challenge has shown that the scientific community is keen to publicly and systematically acknowledge reviewers’ work...

- Philippe Terheggen, Senior Vice President, Physical Sciences II

Since the 1660s, peer review has proved an essential dividing line for judging the difference between science and speculation.

Over the years, however, pressure on the system has continued to grow as the global academic community expands and manuscript submissions rise.

We know that it can be tough for you as Editors to not only find appropriate reviewers willing to review for your journal, but to motivate them to keep reviewing, time and time again. Together with the research community, we are keen to explore ways to ease that pressure and are currently working on a number of peer-review pilots and innovations. One avenue we recently explored was a global competition seeking researchers’ thoughts on the future of peer review. This was inspired by the success of similar competitions such as the Elsevier Grand Challenge and the Executable Paper Grand Challenge.

In an issue of Reviewers’ Update earlier this year, we threw down the gauntlet to readers, asking them to submit ideas that could significantly add to the current peer-review system. We also invited entries that explored how Publishers and Editors can help early career researchers become reviewers, or how reviewers can be recognized by either their institutes or Publishers.

More than 800 readers took up the challenge and entries embraced all aspects of peer review, from improvements to the actual ‘mechanics’ of the process to how to reward reviewers. It was interesting to note that a great many entries detailed suggestions for rewarding reviewers in ways that can be noted on CVs. In addition, there was an intriguing dichotomy between entries that were firmly in favour of the extension of double blind type peer-review systems and those who advocated completely open peer review. The judges opens in new tab/window chose a list of finalists, whose ideas were published on the Peer Review Challenge website opens in new tab/window with an invitation to the academic community to post their comments. More than 300 of you responded and those comments were taken into account by the judges when making their final decision. Many teleconference calls and emails later, the final three winners were chosen and their winning entries can be found below.

Philippe Terheggen, Senior Vice President Physical Sciences II, sat on the judging panel, and was impressed by the quality of the entries. “Thank you to everyone who took the time to enter, or provide us with their feedback. We were really pleased by the number of entries, as well as the thought and initiative invested in them. The next stage is to work with our winners to develop their ideas. We need to determine whether they can be integrated into existing systems, like the peer review annotation idea, or develop the framework necessary to manage and deliver the reviewer points system in a logical way.”

He added: “This Challenge has shown that the scientific community is keen to publicly and systematically acknowledge reviewers’ work, as well as to utilize platforms that make reviewing easier and more transparent for all players. It is our task as Publishers to work with the community, and your task as Editors to make sure that these needs are met.”

If you would like to get involved in any of the pilots mentioned below, please do let me know at [email protected] opens in new tab/window

Overall winner

Elsevier Reviewer Badges and Rewards

Simon Gosling, Lecturer in Physical Geography, Nottingham University, UK

Simon suggested introducing a standardized way to recognize the cumulative effort of a particular reviewer. He explains: “Peer reviews are an important service to the academic community and fundamental to academic publishing. I entered the Challenge because I feel strongly that reviewers should be recognized and rewarded in some way for the time and effort they invest – often on top of a very busy schedule – in preparing article reviews for journal Editors. I see the future of peer review as a positive feedback cycle whereby reviewers (especially early career researchers) – encouraged by an opportunity to enhance their reputation as a reliable and good reviewer and to receive journal and/or book discounts – prepare high quality helpful reviews that in turn help to make the job of journal Editors more straightforward. This way, authors, Editors and reviewers all benefit from the peer-review process. To date, reviewers have received little benefit, let alone anything tangible. I envisage the acknowledgment of reviewers’ time and efforts through a widely recognized tangible accrediting system that would be well-known and citeable on a CV, which I have termed “Elsevier Reviewer Badges”.

Visit the Peer Review Challenge website opens in new tab/window for further details of Simon’s winning entry.

Runner up

Top Reviewer Incentives

Michael Muthukrishna, Graduate student, Department of Psychology, University of British Columbia, Canada

Michael’s idea also concentrates on reviewer rewards and develops the idea of a public points system for reviewers, inspired by websites such as Reddit and Slashdot. Michael elaborates: “As a researcher and a technologist, I was thrilled to hear about Elsevier’s Peer Review Challenge. Technology affords new and often better ways of approaching old problems. The peer-review process consists of established and largely successful practices that support good science and I am cognizant that caution is required when tampering with it. Nevertheless, I see the peer-review process gradually taking advantage of the advancements tested in public, peer-reviewed forums, such as Reddit, Slashdot.org, StackOverflow, Amazon, and many open source software projects. The successful processes in these domains have converged on solutions supported by psychological research, including the power of reputation, which I focused on in my entry.”

Visit the Peer Review Challenge website opens in new tab/window for further details of Michael’s entry.

Runner up

Peer Review Annotation Application

Koen Hufkens, Postdoctoral research associate, Faculty of Bioscience Engineering, Ghent University, Belgium

In contrast to the previous ideas, Koen’s entry suggests that the online platform facilitating peer review could be improved. He suggests creating an integrated review application, which would, among other things, include an integrated PDF reader with annotation possibilities and generate a review template listing the changes to be made based upon annotations on the pdf. This would save reviewers’ time. Of his idea, Koen says: “As a scientist, reviewing for journals is considered an integral part of your work. However, given the voluntary nature of this task, often little time is budgeted towards it. Talking to colleagues it became apparent to me that reviewing for journals is something done when one has a few minutes to spare, e.g. on a flight, train or bus to the next meeting or commuting home. I entered the competition with the idea that a lot of time could be saved by integrating some of today's technology into a reviewing platform.

This platform would integrate reading and annotating on the original document, summarizing these 'in line' notes into a concise summary for the authors to read. Ideally this platform would be online and cloud-based so you could easily pick up and continue where you left of the day before. It's easy to perceive how trends such as tablet applications could extend from this basic idea to provide even greater flexibility and ease of use for the reviewer on the go, or on the couch. As a good work-life balance is limited by available free time, I feel that providing the tools to free up extra time should be considered a major incentive for most scientists to continue reviewing for Elsevier.”

Visit the Peer Review Challenge website opens in new tab/window for further details of Koen’s entry.

Note from Editor

Koen’s ideas chime perfectly with current plans for Evise, Elsevier’s next-generation editorial system. The roll-out of Evise will begin in the second half of 2013 and will include online viewing and annotation of submissions. Reviewers and Editors will be able to view the manuscript and add comments to it online. These comments will then be saved with the submission and can be made available to the author. Koen’s winning entry will be shared with the Evise team.

Contributor