RQC: Review Quality Collector

Lutz Prechelt, prechelt@inf.fu-berlin.de

Review Quality Collector (RQC) is a procedure to be introduced into the scientific publishing system, by which researchers can be formally recognized for their performance as peer reviewers (initially for journals and conferences, later perhaps also for funding agencies).

The first part of its software implementation is online since September 2016. Read up on the concept there:

https://reviewqualitycollector.org

This page (which was once quite long) has now been cut down to the not-yet-quite-outdated parts which may or may not be of interest, but certainly only after you visited https://reviewqualitycollector.org.

---

Related initiatives

This section collects (very very incompletely) links to other initiatives aiming at improving the reviewing process in various ways. If you know of good such material, please drop me a note (with a web link) to prechelt at inf.fu-berlin.de.

Various issues of review quality

• JAMA 280(3), July 1998, had a number of studies related to reviewing quality

Providing review quality feedback to reviewers

• Callaham et al. (2002) found in a controlled experiment that providing basic quality feedback (just a 1-to-5 point rating) confidentially from the editor to average and below-average reviewers did not improve the quality of their subsequent reviews.
• (see also some of the entries under "Recognizing reviewers' work publicly" below)
• Furthermore,
  • many journals send all reviewers of a paper a copy of the editor's reply to the authors (including the text of all reviews).

Recognizing reviewers' work publicly

• Publons is a service for counting and for (if you want to and are allowed to) publishing your reviews. Journals can also nominate reviews as "good", but the grounds for doing so are not transparent and not all journals use the possibility.
• ReviewerCredits is another service for publicly counting your reviews. Its core is transferring "credits" to reviewers that can be redeemed for services. Some journals participate in the RCI index based on review length, speed, and agreement with editor, which reviewers can then publish in their profile.
• "Reviewer of the Year" certificate (since 1994) in MIS Quarterly; list of "outstanding reviewers" for several journals at ACSE; etc.
• Elsevier's Reviewer Recognition Platform: Implements Simon Gosling's winning entry in the 2012 Peer Review Challenge to collect ideas for improving the peer review system: The Reviewer Badges and Rewards scheme which proposes to publicly recognize reviewers for the quantity of their reviewing.
  In 2013, Elsevier issued certificates to the 25 best reviewers of each of a number of journals (where "best" was decided informally by each editor).
  They are now intending to do both of these or similar steps regularly for more and more journals.
• Peerage of Science even has a simple review quality rating as a kind of byproduct.
• Furthermore,
  • A Science Magazine article in Feb 2016 discussed such initiatives.
  • Many journals publish a list of all their reviewers at the end of each year.
  • …

In particular: Open peer review (OPR)

• Preprint commentary/open peer review/post publication peer review venues (Google Doc)
• F1000Research is an open-access journal founded by Vitek Tracz, the inventor of BioMed Central. It performs post-publication open reviewing (and commenting and author-reviewer dialog and article revisions):
  "All submissions to F1000Research will go through a rapid initial check by the in-house editorial team before being published with the status 'Awaiting Peer Review'."
  Reviewing results become part of the article metadata, only positively reviewed articles are indexed.
• ScienceOpen also uses a publication style of post-publication open reviewing and commenting.
• A few journals (sometimes) publish the reviewer reports along with accepted articles, for instance
  • BMJ
  • Agricultural and Forest Meteorology (according to Editor's Update 36(part2), 2012, Section "Published reviewer reports")
  • Some PLoS journals additionally make some comments publicly made on preprints of the submissions a part of the review history
  • …
• Nature trialed "open peer review" in 2006: Authors were asked to agree to post their submitted articles publicly for receiving non-anonymous and public review comments.
  Only 5% of the authors agreed to participate with their article. Only 54% of those articles received any comment at all (most of these only one). The editors judged only 4% of the comments to have the quality of normal reviews.
• Frontiers uses a format called "collaborative peer review" and has very detailed review process guidelines.
• According to the "What do we know about peer review" presentation on the BMJ reviewer resources page, three controlled trials found that signing a review did not improve its quality. (But the references are vague: Godlee et al, 1998; van Rooyen et al, 1998; van Rooyen et al, 1999)
• Blogosphere: A lot is written about open peer review in various blogs.
  ScienceOpen collects such material in the ScienceOpen Blog and on the ScienceOpen facebook page.
  Or see these good articles:
  • Alexander Grossmann (ScienceOpen): Where did our Peer Review Mojo go? (on anomyity, incentives, reward, etc.)
  • Lenny Teytelman (PubChase): We Can Fix Peer Review Now
  • Hilda Bastian (PLoS): Signing Critical Peer Reviews & the Fear of Retaliation: What Should We Do?
  • …
• Open Science:
  • OpenUP Hub's Open Peer Review Toolbox
  • FOSTER's open science online course on Open Peer Review
  • PublishYourReviews, a pledge initiative to publish one's reviews if a preprint of the reviewed submission is available

Other

• CWTS course on responsible metrics in research evaluation (2019), refers to the Leiden Manifesto.
• PRE-Score assigns a quantitative score to a journal based on various characteristics of that journal's reviewing process, with the intention of allowing authors, readers, and libraries to identify journals where submitted work is "properly vetted".
• In Spain, in order to get promoted to a professorship, a researcher needs to bring a letter attesting his or her reviews from each journal for which the researcher claims to have reviewed.
  These are collected by a central agency ANECA responsible for nationwide researcher quality assessment.
• Furthermore,
  • some journals let reviewers pick manuscripts for review (rather than have an editor ask them to review a particular one), for instance
    • Journal of Universal Computer Science
    • Chemical Physics Letters (according to Editor's Update 36(part2), 2012, Section "PeerChoice")
  • Nature hosted an online debate on peer review in 2006 that features a number of interesting position papers.
• Casrai.org standardizes research administration information
• ASAPbio proposes to separate the evaluation function of peer review from the gatekeeping function by installing a peer feedback step before the journal review step.
• ORCID peer review early adopter program, peer review workflow

---

Related research

• Are non-monetary rewards effective in attracting peer reviewers? A natural experiment (Zaharie and Seeber 2018). Shows that review quality evaluation can damage intrinsic motivation if framed wrongly. (Link)
• A review article about research on peer review (Birukou et al. 2011)
• A fixed questionnaire, called RQI, to serve as a measurement instrument for review quality (van Rooyen et al. 1999)
• "Many [reviewers] wanted to see feedback on their own performance, perhaps an end of year review.
  Individuals who had received an acknowledgement or some form of feedback in the past implied they were more pre-disposed to review for that journal." (Mulligan, 2004)
• "It was felt that many of the perceived problems afflicting the refereeing process could be remedied if the review was formally recognised.
  Such a system could take the form of accreditation to a journal, society or publisher. It might identify the number of times an individual reviews in different prestigious journals; such an indicator could in part be a measure of that individual's contribution to science." (Mulligan, 2004)
  (RQC covers this idea -- and goes far beyond it.)
• A survey (sense about science, 2009) and focus groups (Mulligan, 2004) found that
  • The dominant motivators for reviewers today are participating in the scientific process (whether for the joy of it or out of a sense of duty), helping the authors, and interest in the content of the articles.
    (So RQC can create an additional source of motivation.)
  • Editors find it becomes harder and harder to find and keep good reviewers.
  • Editors perceive the nature and quality of the reviews they receive as highly fluctuating.
  • And many other interesting attitudes.
• see also the Callaham entry under "Providing review quality feedback" above
• For an overview of various experimental studies on improving peer review in the biomedical field, see Jefferson et al. (2008).
• For some fundamental thoughts on what constitutes peer review quality, see Charles Jennings' Quality and value: The true purpose of peer review?
• The online-consultation "scientific publication system" generated some comments on reviewing: too many publications, often sloppy reviews. See Section "Results 2", point 6 in this summary.
• A survey of senior authors and reviewers determined various attitudes towards reviewing, the indicate predominantly (64%) satisfaction, but also the feeling that improvements are possible (only 32% think they are not). (Ware 2009)
• Publons' "Global State of Peer Review" and a summary in Nature News
• 2015 Springer survey on reviewer motivation, 1400 responses: "Reviewers were generally against a points-based system or rewards based on number of reviews and instead preferred being rewarded and recognized based on the quality of the review"

References:

• http://asapbio.org/six-essential-reads-on-peer-review, 2018-01
• Ross-Hellauer, Deppe, Schmidt: Survey on open peer review: Attitudes and experience amongst editors, authors and reviewers, 2017-12.
  Only 44% believe open identities will help review quality; 60% ditto for open reports.
• Aliaksandr Birukou, Joseph Rushton Wakeling, Claudio Bartolini, Fabio Casati, Maurizio Marchese, Katsiaryna Mirylenka, Nardine Osman, Azzurra Ragone, Carles Sierra, Aalam Wassef, 2011: Alternatives to peer review: novel approaches for research evaluation
• Michael L. Callaham, Robert K. Knopp, E. John Gallagher, 2002: Effect of Written Feedback by Editors on Quality of Reviews: Two Randomized Trials
  J. American Medial Association 287(21):2781-2783
• T. Jefferson, M. Rudin, S. Brodney Folse, F. Davidoff, 2008: Editorial Peer Review for Improving the Quality of Reports of Biomedical Studies
  Cochrane Database of Systematic Reviews, John Wiley & Sons.
• Adrian Mulligan, 2004: Is Peer Review in Crisis?
  Perspectives in Publishing, August.
• Sense About Science, 2009: Peer Review Survey 2009.
• Susan van Rooyen, Nick Black, Fiona Godlee, 1999: Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts, Journal of Clinical Epidemiology, 52(7):625-629.
• M. Ware, 2008: Peer review in scholarly journals: Perspective of the scholarly community-Results from an international study., Information Services & Use 28:109-112.
• Bezis, Birukou: Arbitrariness in the peer review process, Scientometrics, 2020

---

Other peer-review stuff

• The Rise of Peer Review: Peer review became mainstream only in the late 1970s!
• John Bohannon: "Who's afraid of peer review?: A spoof paper concocted by Science reveals little or no scrutiny at many open-access journals", Science 342(6154):60-65, Oct 4 2013.
• The Science Open blog had a 2016-02 series of posts on advances in peer review: http://blog.scienceopen.com/2016/02/advances-in-peer-review/
• Peerage of Science is a service that central-sources reviews: Reviewers sign up for submissions, provide reviews, and these reviews become available to all participating journals, which can then react.
• TheScientist: How to Make Scientists Into Better Peer Reviewers
• Ioannidis: How to make more published research true, article about problems in our reward system and how to align it
• Guardian article by James Heathers about a retraction in The Lancet of an article about a CoViD19 drug.
• LSEImpactBlog: The four schools of thought on reforming peer review
• Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability
• NISO peer review terminology

---

Research assessment

• https://sfdora.org/
• https://responsiblemetrics.org/wp-content/uploads/2019/02/2015_metrictide.pdf report (UK, 2015-07, lead author James Wilsdon) about responsible vs. irresponsible ways of using metrics. See https://responsiblemetrics.org and summary on https://re.ukri.org/sector-guidance/publications/metric-tide/.

Loosely related things

• Plaudit connects an ORCID ID and an endorsement with a DOI.
• Björn Brembs about the cost of publishing 1 article.
• APE (Academic Publishing Europe) is a conference of mostly the publishing industry. APE 2019
• Force11 "is a community of scholars, librarians, archivists, publishers and research funders that has arisen organically to help facilitate the change toward improved knowledge creation and sharing [...] through the effective use of information technology."
• …