Office of Management and Budget

  • Regulation & Information
  • Intellectual Property
  • Legislative

M-05-03, Peer Review

Image of OMB Letterhead

December 16, 2004

MEMORANDUM FOR HEADS OF DEPARTMENTS AND AGENCIES

OMB has today issued a bulletin applicable to all departments and agencies entitled "Final Information Quality Bulletin for Peer Review." This Bulletin establishes government-wide guidance aimed at enhancing the practice of peer review of government science documents. Peer review is an important procedure used by the scientific community to ensure that the quality of published information. Peer review can increase the quality and credibility of the scientific information generated across the federal government. This Bulletin is one aspect of a larger OMB effort to improve the quality of the scientific information upon which policy decisions are based.

The bulletin has benefited from extensive public and agency comments received on two prior draft versions, which were released by OMB in September 15, 2003 and April 28, 2004. The bulletin includes guidance to federal agencies on what information is subject to peer review, the selection of appropriate peer reviewers, opportunities for public participation, and related issues. The bulletin also defines a peer review planning process that will permit the public and scientific societies to contribute to agency dialogue about which scientific reports merit especially rigorous peer review.

If your staff has questions about this guidance, please contact Margo Schwab at (202) 395-5647 or [email protected] .

How to Write a Peer Review: 12 things you need to know

peer review 03

Joanna Wilkinson

Learning how to peer review is no small feat. You’re responsible for protecting the public from false findings and research flaws, while at the same time helping to uncover legitimate breakthroughs. You’re also asked to constructively critique the research of your peers, some of which has taken blood, sweat, tears and years to put together.

Despite this, peer review doesn’t need to be hard or nerve-wracking–or make you feel like you’re doomed to fail.

We’ve put together  12 tips to help with peer review , and you can learn the entire process with our free peer review training course, the  Web of Science Academy . This on-demand, practical course and comes with one-to-one support with your own mentor. You’ll have exclusive access to our peer review template, plenty of expert review examples to learn from, and by the end of it, you’ll not only be a certified reviewer, we’ll help put you in front of editors in your field.

The peer review process

Journal peer review is a critical tool for ensuring the quality and integrity of the research literature. It is the process by which researchers use their expert knowledge of a topic to assess an article for its accuracy and rigor, and to help make sure it builds on and adds to the current literature.

It’s actually a very structured process; it can be learned and improved the more you do it, and you’ll become faster and more confident as time goes on. Soon enough, you’ll even start benefiting from the process yourself.

Peer review not only helps to maintain the quality and integrity of literature in your field, it’s key to your own development as a researcher. It’s a great way to keep abreast of current research, impress editors at elite journals, and hone your critical analysis skills. It teaches you how to  review a manuscript ,  spot common flaws in research papers , and improve your own chances of being a  successful published author .

12-step guide to writing a peer review

To get the most out of the peer review process, you’ll want to keep some best practice tips and techniques in mind from the start. This will help you write a review around two to three pages (four maximum) in length.

We asked an expert panel of researchers what steps they take to ensure a thorough and robust review. We then compiled their advice into 12 easy steps with link to blog posts for further information:

1)   Make sure you have the right expertise.  Check out our post,  Are you the right reviewer?  for our checklist to assess whether you should take on a certain peer review request.

2)   Visit the journal web page to learn their reviewer-specific instructions.  Check the manuscript fits in the journal format and the references are standardised (if the editor has not already done so).

3)   Skim the paper very quickly to get a general sense of the article.  Underline key words and arguments, and summarise key points. This will help you quickly “tune in” to the paper during the next read.

4)   Sit in a quiet place and read the manuscript critically.  Make sure you have the tables, figures and references visible. Ask yourself key questions, including: Does it have a relevant title and valuable research question? Are key papers referenced? What’s the author’s motivation for the study and the idea behind it? Are the data and tools suitable and correct? What’s new about it? Why does that matter? Are there other considerations? Find out more in our  12-step guide to critically reviewing a manuscript .

5)   Take notes about the major, moderate and minor revisions that need to be made . You need to make sure you can put the paper down and come back to it with fresh eyes later on. Note-taking is essential for this.

6)   Are there any methodological concerns or common research errors?  Check out our guide for  common research flaws to watch out for .

7)   Create a list of things to check.  For example, does the referenced study actually show what is claimed in the paper?

8)   Assess language and grammar, and make sure it’s a right ‘fit’ for the journal.  Does the paper flow? Does it have connectivity? Does it have clarity? Are the words and structure concise and effective?

9)   Is it new research?  Check previous publications of the authors and of other authors in the field to be sure that the results were not published before.

10)   Summarise your notes for the editor.  This can include overview, contribution, strengths & weaknesses, and acceptability. You can also include the manuscript’s contribution/context for the authors (really just to clarify whether you view it similarly, or not), then prioritise and collate the major revisions and minor/specific revisions into feedback. Try to compile this in a logical way, grouping similar things under a common heading where possible, and numbering them for ease of reference.

11)   Give specific recommendations to the authors for changes.  What do you want them to work on? in the manuscript that the authors can do.

12)  Give your recommendation to the editor.

We hope these 12 steps help get you on your way for your first peer review, or improving the structure of your current reviews. And remember, if you’d like to master the skills involved in peer review and get access to our Peer Review Template, sign up for our  Web of Science Academy .

Our expert panel of reviewers include:  Ana Marie Florea  (Heinrich-Heine-Universität Düsseldorf),  James Cotter  (University of Otago), and  Robert Faff  (University of Queensland). These reviewers are all recipients of the Global Peer Review Awards powered by Publons. They also and boast hundreds of pre-publication peer reviews for more than 100 different journals and sit on numerous editorial boards.

Related posts

2024 journal citation reports: changes in journal impact factor category rankings to enhance transparency and inclusivity.

peer review 03

New Web of Science Grants Index helps researchers develop more targeted grant proposals

peer review 03

Three ways research offices can lead researchers to more funding

peer review 03

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

  • PLOS Biology
  • PLOS Climate
  • PLOS Complex Systems
  • PLOS Computational Biology
  • PLOS Digital Health
  • PLOS Genetics
  • PLOS Global Public Health
  • PLOS Medicine
  • PLOS Mental Health
  • PLOS Neglected Tropical Diseases
  • PLOS Pathogens
  • PLOS Sustainability and Transformation
  • PLOS Collections

Open Peer Review

A photo of glasses on a desk with a keyboard and open notebook

What is Open Peer Review, and how is it different from other review models?

Peer review is a pillar of scientific communication, the mechanism we rely on to ensure that published research is thoroughly vetted and scientifically valid. For that reason, we tend to think of peer review as a monolith–iconic, stable, and consistent. In fact, journals use many different forms and applications of peer review, often in parallel.

What does it mean to practice Open Peer Review?

That is sort of a trick question…Open Peer Review has many different definitions. It is a general, catch-all term used to describe any peer review model in which aspects of the peer review process are made publicly available, either before or after publication.

For example, Open Peer Review models may include any of the following transparent practices, either alone or in combination:

  • publishing peer review content
  • open commenting from the wider community
  • open discussion between authors, editors and reviewers
  • open review before publication through preprints
  • post-publication commenting
  • sharing author or reviewer identities
  • decoupling the peer review process from the publication process

At least 22 different peer review configurations were already in use across scientific publishing as of 2017 (Ross-Hellauer).

It’s important to treat each attribute of transparent review separately, since each can have a different impact on the author, reviewer, and reader experience. Here, we’ll be talking primarily about publishing peer review content, as well as touching lightly on signed review.

peer review 03

Published Peer Review History at PLOS

Published Peer Review History collects the correspondence exchanged during the peer review possess—including decision letters from each revision, complete with both editorial feedback and peer reviews, and the authors’ responses to reviewers—and makes it available alongside a published research article as a permanent part of the scientific record. PLOS authors can opt-in to publish their Peer Review History once their article is accepted for publication. If the reviewers have chosen to sign their review, their names will appear alongside their comments. If not, the reviews will appear anonymously.

peer review 03

Why is Open Peer Review important?

Depending on your discipline and the length of time you’ve worked in research, you may see Open Peer Review as industry standard, entirely new and novel, or somewhere inbetween.

Open Peer Review in one form or another has been part of the scholarly communications conversation since the 1980s (McGiffert, 1988). Both signed and published peer review have been practiced at The BMJ and BMJ family journals for over 20 years (van Rooyen 1998, 1999), which may partially account for their prevalence in medical and health-related fields. Public review is similarly accepted in computational fields―perhaps because many researchers have a background in coding, and are comfortable with the public commenting common on GitHub and similar code repositories.

In the life sciences, Open Peer Review is less established but is becoming increasingly normalized. For example, journals including Royal Society Open Science, Nature Communications, EMBO, eLife, and of course the PLOS journals, all offer forms of Open Peer Review.

No matter the focus of your research, the odds are good that you’ll be invited to participate in an Open Peer Review process during your career―if you haven’t already.

The value of published peer review

Proponents of published review argue that increasing transparency in the peer review process leads to a better understanding of published research, more constructive peer reviews, and well-deserved credit for reviewers.

✔ Enriching the scientific record

Published peer review helps contextualize research and gives readers the benefit of additional expert opinions. Letting readers see the questions reviewers raised and how the authors mitigated them gives insight into the limits of the study. Publishing peer reviews also reinforces the validity of the article by exposing the rigorous vetting process it has undergone prior to publication.

✔ Honoring reviews and reviewers

Choosing to publish peer reviews acknowledges the value of review, and the vital role that it plays in scientific communication by making it a permanent part of the scientific record. Offering reviewers the opportunity to sign is just one of several ways for them to claim credit for their work.

✔ Educational tools

Making peer review publicly available creates a database of examples for students to reference as they begin to participate in peer review.

✔ Quality of feedback

Research suggests reviews with the potential to be published tend to be at least as good as, and maybe slightly better than, reviews which remain private. Two BMJ studies on signed review (Van Rooyen, 2001) and impact of possible public posting (van Rooyen, 2010) found no difference in quality; however, other research found improvements in specific areas like constructive feedback, comments on methods, length of review, and substantiating evidence to support the comments (Kowalczuk, 2013; Walsh, 2000; Bornmann, 2012; Mehmani, 2016).

Concerns about published peer review

✘ Diminishing reviewer capacity

Just as telling reviewers their comments may be published can improve the quality of the review, some detractors argue that publishing reviews puts undue pressure on reviewers. For example, knowing that a peer review could be published may lead reviewers to spend extra time on spelling or grammar―aspects of the review which really don’t impact the science, but which can be time consuming, especially for researchers who primarily use other languages. That could contribute to a more demanding and stressful experience for reviewers, and less time for review, and more declined invitations.

On signed peer review

Revealing reviewer identities to authors helps to contextualize the feedback, providing insight into potential competing interests, and emphasizing once again the rigor, thoroughness, and appropriateness of the peer review assessment. Signing also gives reviewers an opportunity to claim academic credit for their contributions to the study. That potential only increases when signing operates in tandem with published review.

At the same time, revealing reviewer identities is by far the most common reason stakeholders object to openness in peer review. Many in the research community worry about the threat of negative career consequences for critical reviewers, especially for junior researchers who are dependent on senior scientists for opportunities and advancement.

Harassment, bullying, and blackballing do occur in science–and even anonymous review is not necessarily protection. According to some studies, authors can guess the identities of their reviewers in as many as 10% (van Rooyen, 1998) to 32% (Justice, 1998) of cases. Even if the authors are wrong about the identities of their reviewers, that doesn’t necessarily mean that career damage won’t occur―for someone. These concerns are valid, and those of us on the periphery of research, like publishers, funders, and administrators, often can’t know the political and social circumstances in which researchers are operating. For that reason, it’s important that individual reviewers have the flexibility to decide whether to sign their name to a review in each specific case.

Meanwhile, journals, funders and institutions can have a positive impact by explicitly naming retribution against peer reviewers as misconduct with consequences (Bastian, 2018), and by making the whole process more transparent and open to public scrutiny, thereby reducing opportunities to engage in negative behaviors.

Have your say

How do i complete an “open,” signed, or published peer review.

First, decide where you stand. Take some time to review the research, talk to colleagues you respect, and consider whether you want to sign your reviews as a general principle. Remember, just because you have a personal philosophy doesn’t mean there won’t be occasional exceptions–but at least this way you’ll feel confident about your choices and the reasons behind them.

The practical aspects of completing a peer review remain the same, regardless of the review model a particular journal might happen to employ. There are three key stages to completing a peer review, Open or otherwise.

  • Responding to invitations. Accept an invitation to review only if you have the necessary time and expertise, and can provide an objective review. Journals will let you know what peer review model they practice before you agree to review, usually in the invitation letter itself, or on their website. Read more.
  • Reading the manuscript. Before you read the manuscript, make sure that you understand the journal’s criteria, so you know what to watch for as you make your assessment. Read the manuscript through once for a general understanding, and a second time taking notes as you go. Pay special attention to the research question, methods, and conclusions, and review the figures and tables in conjunction with the results. Read more .
  • Writing the Review. Begin with a summary of the research and your overall impression, before moving on to discuss specific areas for improvement and any other points. There is no need to do the authors’ work for them by suggesting line edits, specific manuscripts to cite, or experiments to do. Just let the editors know where you had questions or concerns. Read more.

Bastian, H. (2018, March 22). Signing Critical Peer Reviews & the Fear of Retaliation: What Should we do? [Blog post]. Absolutely Maybe.

Bolam, P. (2017, September 14). Transparent Review at the European Journal of Neuroscience: Experiences One Year On , [Blog post].

Bornmann, L., Wolf, M. & Daniel, H.D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91 (3): 843-856.

Justice A.C., Cho M.K., Winker M.A., Berlin J.A., Rennie D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial , JAMA. 280 (3):240-2.

Kowalczuk, M.K , Dudbridge, F. , Nanda, S., Harriman, S.L., & Moylan, E.C. (2013). A comparison of the quality of reviewer reports from author-suggested reviewers and editor-suggested reviewers in journals operating on open or closed peer review models . F1000Research .

McGiffert M. (1988). Is Justice Blind? An Inquiry into Peer Review. Scholarly Publishing, 20(1): 43-48.

Mehmani, B. (2016, September 22). Is Open Peer Review the Way Forward? [Blog].

Pulverer, B. (2010). A transparent black box. The EMBO Journal, 29 (23):3891-3892.

Ross-Hellauer, T. (2017). What is open peer review? A systematic review [version 2; peer review: 4 approved]. F1000Research, 6 :588 

van Rooyen, S., Godlee, F., Evans, S., Smith, R., Black, N. (1998) Effect of blinding and unmasking on the quality of peer review: a randomized trial . JAMA , 280 (3):234-7.

van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999, Jan 2). Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ, 318 (7175):23-7.

van Rooyen, S., Delamothe, T. & Evans, S.J. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial . BMJ, 341 :5729. 

Walsh, E. Rooney, M. & Wilinson, G. (2000). Open peer review: a randomised controlled trial . British Journal of Psychiatry, 176 :47-51.

Disclaimer » Advertising

  • HealthyChildren.org

Issue Cover

  • Previous Article
  • Next Article

What is the Purpose of Peer Review?

What makes a good peer reviewer, how do you decide whether to review a paper, how do you complete a peer review, limitations of peer review, conclusions, research methods: how to perform an effective peer review.

  • Split-Screen
  • Article contents
  • Figures & tables
  • Supplementary Data
  • Peer Review
  • CME Quiz Close Quiz
  • Open the PDF for in another window
  • Get Permissions
  • Cite Icon Cite
  • Search Site

Elise Peterson Lu , Brett G. Fischer , Melissa A. Plesac , Andrew P.J. Olson; Research Methods: How to Perform an Effective Peer Review. Hosp Pediatr November 2022; 12 (11): e409–e413. https://doi.org/10.1542/hpeds.2022-006764

Download citation file:

  • Ris (Zotero)
  • Reference Manager

Scientific peer review has existed for centuries and is a cornerstone of the scientific publication process. Because the number of scientific publications has rapidly increased over the past decades, so has the number of peer reviews and peer reviewers. In this paper, drawing on the relevant medical literature and our collective experience as peer reviewers, we provide a user guide to the peer review process, including discussion of the purpose and limitations of peer review, the qualities of a good peer reviewer, and a step-by-step process of how to conduct an effective peer review.

Peer review has been a part of scientific publications since 1665, when the Philosophical Transactions of the Royal Society became the first publication to formalize a system of expert review. 1 , 2   It became an institutionalized part of science in the latter half of the 20 th century and is now the standard in scientific research publications. 3   In 2012, there were more than 28 000 scholarly peer-reviewed journals and more than 3 million peer reviewed articles are now published annually. 3 , 4   However, even with this volume, most peer reviewers learn to review “on the (unpaid) job” and no standard training system exists to ensure quality and consistency. 5   Expectations and format vary between journals and most, but not all, provide basic instructions for reviewers. In this paper, we provide a general introduction to the peer review process and identify common strategies for success as well as pitfalls to avoid.

Modern peer review serves 2 primary purposes: (1) as “a screen before the diffusion of new knowledge” 6   and (2) as a method to improve the quality of published work. 1 , 5  

As screeners, peer reviewers evaluate the quality, validity, relevance, and significance of research before publication to maintain the credibility of the publications they serve and their fields of study. 1 , 2 , 7   Although peer reviewers are not the final decision makers on publication (that role belongs to the editor), their recommendations affect editorial decisions and thoughtful comments influence an article’s fate. 6 , 8  

As advisors and evaluators of manuscripts, reviewers have an opportunity and responsibility to give authors an outside expert’s perspective on their work. 9   They provide feedback that can improve methodology, enhance rigor, improve clarity, and redefine the scope of articles. 5 , 8 , 10   This often happens even if a paper is not ultimately accepted at the reviewer’s journal because peer reviewers’ comments are incorporated into revised drafts that are submitted to another journal. In a 2019 survey of authors, reviewers, and editors, 83% said that peer review helps science communication and 90% of authors reported that peer review improved their last paper. 11  

Expertise: Peer reviewers should be up to date with current literature, practice guidelines, and methodology within their subject area. However, academic rank and seniority do not define expertise and are not actually correlated with performance in peer review. 13  

Professionalism: Reviewers should be reliable and objective, aware of their own biases, and respectful of the confidentiality of the peer review process.

Critical skill : Reviewers should be organized, thorough, and detailed in their critique with the goal of improving the manuscript under their review, regardless of disposition. They should provide constructive comments that are specific and addressable, referencing literature when possible. A peer reviewer should leave a paper better than he or she found it.

Is the manuscript within your area of expertise? Generally, if you are asked to review a paper, it is because an editor felt that you were a qualified expert. In a 2019 survey, 74% of requested reviews were within the reviewer’s area of expertise. 11   This, of course, does not mean that you must be widely published in the area, only that you have enough expertise and comfort with the topic to critique and add to the paper.

Do you have any biases that may affect your review? Are there elements of the methodology, content area, or theory with which you disagree? Some disagreements between authors and reviewers are common, expected, and even helpful. However, if a reviewer fundamentally disagrees with an author’s premise such that he or she cannot be constructive, the review invitation should be declined.

Do you have the time? The average review for a clinical journal takes 5 to 6 hours, though many take longer depending on the complexity of the research and the experience of the reviewer. 1 , 14   Journals vary on the requested timeline for return of reviews, though it is usually 1 to 4 weeks. Peer review is often the longest part of the publication process and delays contribute to slower dissemination of important work and decreased author satisfaction. 15   Be mindful of your schedule and only accept a review invitation if you can reasonably return the review in the requested time.

Once you have determined that you are the right person and decided to take on the review, reply to the inviting e-mail or click the associated link to accept (or decline) the invitation. Journal editors invite a limited number of reviewers at a time and wait for responses before inviting others. A common complaint among journal editors surveyed was that reviewers would often take days to weeks to respond to requests, or not respond at all, making it difficult to find appropriate reviewers and prolonging an already long process. 5  

Now that you have decided to take on the review, it is best of have a systematic way of both evaluating the manuscript and writing the review. Various suggestions exist in the literature, but we will describe our standard procedure for review, incorporating specific do’s and don’ts summarized in Table 1 .

Dos and Don’ts of Peer Review

First, read the manuscript once without making notes or forming opinions to get a sense of the paper as whole. Assess the overall tone and flow and define what the authors identify as the main point of their work. Does the work overall make sense? Do the authors tell the story effectively?

Next, read the manuscript again with an eye toward review, taking notes and formulating thoughts on strengths and weaknesses. Consider the methodology and identify the specific type of research described. Refer to the corresponding reporting guideline if applicable (CONSORT for randomized control trials, STROBE for observational studies, PRISMA for systematic reviews). Reporting guidelines often include a checklist, flow diagram, or structured text giving a minimum list of information needed in a manuscript based on the type of research done. 16   This allows the reviewer to formulate a more nuanced and specific assessment of the manuscript.

Next, review the main findings, the significance of the work, and what contribution it makes to the field. Examine the presentation and flow of the manuscript but do not copy edit the text. At this point, you should start to write your review. Some journals provide a format for their reviews, but often it is up to the reviewer. In surveys of journal editors and reviewers, a review organized by manuscript section was the most favored, 5 , 6   so that is what we will describe here.

As you write your review, consider starting with a brief summary of the work that identifies the main topic, explains the basic approach, and describes the findings and conclusions. 12 , 17   Though not universally included in all reviews, we have found this step to be helpful in ensuring that the work is conveyed clearly enough for the reviewer to summarize it. Include brief notes on the significance of the work and what it adds to current knowledge. Critique the presentation of the work: is it clearly written? Is its length appropriate? List any major concerns with the work overall, such as major methodological flaws or inaccurate conclusions that should disqualify it from publication, though do not comment directly on disposition. Then perform your review by section:

Abstract : Is it consistent with the rest of the paper? Does it adequately describe the major points?

Introduction : This section should provide adequate background to explain the need for the study. Generally, classic or highly relevant studies should be cited, but citations do not have to be exhaustive. The research question and hypothesis should be clearly stated.

Methods: Evaluate both the methods themselves and the way in which they are explained. Does the methodology used meet the needs of the questions proposed? Is there sufficient detail to explain what the authors did and, if not, what needs to be added? For clinical research, examine the inclusion/exclusion criteria, control populations, and possible sources of bias. Reporting guidelines can be particularly helpful in determining the appropriateness of the methods and how they are reported.

Some journals will expect an evaluation of the statistics used, whereas others will have a separate statistician evaluate, and the reviewers are generally not expected to have an exhaustive knowledge of statistical methods. Clarify expectations if needed and, if you do not feel qualified to evaluate the statistics, make this clear in your review.

Results: Evaluate the presentation of the results. Is information given in sufficient detail to assess credibility? Are the results consistent with the methodology reported? Are the figures and tables consistent with the text, easy to interpret, and relevant to the work? Make note of data that could be better detailed in figures or tables, rather than included in the text. Make note of inappropriate interpretation in the results section (this should be in discussion) or rehashing of methods.

Discussion: Evaluate the authors’ interpretation of their results, how they address limitations, and the implications of their work. How does the work contribute to the field, and do the authors adequately describe those contributions? Make note of overinterpretation or conclusions not supported by the data.

The length of your review often correlates with your opinion of the quality of the work. If an article has major flaws that you think preclude publication, write a brief review that focuses on the big picture. Articles that may not be accepted but still represent quality work merit longer reviews aimed at helping the author improve the work for resubmission elsewhere.

Generally, do not include your recommendation on disposition in the body of the review itself. Acceptance or rejection is ultimately determined by the editor and including your recommendation in your comments to the authors can be confusing. A journal editor’s decision on acceptance or rejection may depend on more factors than just the quality of the work, including the subject area, journal priorities, other contemporaneous submissions, and page constraints.

Many submission sites include a separate question asking whether to accept, accept with major revision, or reject. If this specific format is not included, then add your recommendation in the “confidential notes to the editor.” Your recommendation should be consistent with the content of your review: don’t give a glowing review but recommend rejection or harshly criticize a manuscript but recommend publication. Last, regardless of your ultimate recommendation on disposition, it is imperative to use respectful and professional language and tone in your written review.

Although peer review is often described as the “gatekeeper” of science and characterized as a quality control measure, peer review is not ideally designed to detect fundamental errors, plagiarism, or fraud. In multiple studies, peer reviewers detected only 20% to 33% of intentionally inserted errors in scientific manuscripts. 18 , 19   Plagiarism similarly is not detected in peer review, largely because of the huge volume of literature available to plagiarize. Most journals now use computer software to identify plagiarism before a manuscript goes to peer review. Finally, outright fraud often goes undetected in peer review. Reviewers start from a position of respect for the authors and trust the data they are given barring obvious inconsistencies. Ultimately, reviewers are “gatekeepers, not detectives.” 7  

Peer review is also limited by bias. Even with the best of intentions, reviewers bring biases including but not limited to prestige bias, affiliation bias, nationality bias, language bias, gender bias, content bias, confirmation bias, bias against interdisciplinary research, publication bias, conservatism, and bias of conflict of interest. 3 , 4 , 6   For example, peer reviewers score methodology higher and are more likely to recommend publication when prestigious author names or institutions are visible. 20   Although bias can be mitigated both by the reviewer and by the journal, it cannot be eliminated. Reviewers should be mindful of their own biases while performing reviews and work to actively mitigate them. For example, if English language editing is necessary, state this with specific examples rather than suggesting the authors seek editing by a “native English speaker.”

Peer review is an essential, though imperfect, part of the forward movement of science. Peer review can function as both a gatekeeper to protect the published record of science and a mechanism to improve research at the level of individual manuscripts. Here, we have described our strategy, summarized in Table 2 , for performing a thorough peer review, with a focus on organization, objectivity, and constructiveness. By using a systematized strategy to evaluate manuscripts and an organized format for writing reviews, you can provide a relatively objective perspective in editorial decision-making. By providing specific and constructive feedback to authors, you contribute to the quality of the published literature.

Take-home Points

FUNDING: No external funding.

CONFLICT OF INTEREST DISCLOSURES: The authors have indicated they have no potential conflicts of interest to disclose.

Dr Lu performed the literature review and wrote the manuscript. Dr Fischer assisted in the literature review and reviewed and edited the manuscript. Dr Plesac provided background information on the process of peer review, reviewed and edited the manuscript, and completed revisions. Dr Olson provided background information and practical advice, critically reviewed and revised the manuscript, and approved the final manuscript.

Advertising Disclaimer »

Citing articles via

Email alerts.

peer review 03

Affiliations

  • Editorial Board
  • Editorial Policies
  • Pediatrics On Call
  • Online ISSN 2154-1671
  • Print ISSN 2154-1663
  • Pediatrics Open Science
  • Hospital Pediatrics
  • Pediatrics in Review
  • AAP Grand Rounds
  • Latest News
  • Pediatric Care Online
  • Red Book Online
  • Pediatric Patient Education
  • AAP Toolkits
  • AAP Pediatric Coding Newsletter

First 1,000 Days Knowledge Center

Institutions/librarians, group practices, licensing/permissions, integrations, advertising.

  • Privacy Statement | Accessibility Statement | Terms of Use | Support Center | Contact Us
  • © Copyright American Academy of Pediatrics

This Feature Is Available To Subscribers Only

Sign In or Create an Account

  • Search Search
  • CN (Chinese)
  • DE (German)
  • ES (Spanish)
  • FR (Français)
  • JP (Japanese)
  • Open research
  • Booksellers
  • Peer Reviewers
  • Springer Nature Group ↗
  • Publish an article
  • Roles and responsibilities
  • Signing your contract
  • Writing your manuscript
  • Submitting your manuscript
  • Producing your book
  • Promoting your book
  • Submit your book idea
  • Manuscript guidelines
  • Book author services
  • Publish a book
  • Publish conference proceedings

How to peer review

Author tutorials 

For science to progress, research methods and findings need to be closely examined and verified, and from them a decision on the best direction for future research is made. After a study has gone through peer review and is accepted for publication, scientists and the public can be confident that the study has met certain standards, and that the results can be trusted.

What you will get from this course

When you have completed this course and the included quizzes, you will have gained the skills needed to evaluate another researcher’s manuscript in a way that will help a journal Editor make a decision about publication. Additionally, having successfully completed the quizzes will let you demonstrate that competence to the wider research community

Topics covered

How the peer review process works.

Journals use peer review to both validate the research reported in submitted manuscripts, and sometimes to help inform their decisions about whether or not to publish that article in their journal. 

If the Editor does not immediately reject the manuscript (a “desk rejection”), then the editor will send the manuscript to two or more experts in the field to review it. The experts—called peer reviewers—will then prepare a report that assesses the manuscript, and return it to the editor. After reading the peer reviewer's report, the editor will decide to do one of three things: reject the manuscript, accept the manuscript, or ask the authors to revise and resubmit the manuscript after responding to the peer reviewers’ feedback. If the authors resubmit the manuscript, editors will sometimes ask the same peer reviewers to look over the manuscript again to see if their concerns have been addressed. This is called re-review.

Some of the problems that peer reviewers may find in a manuscript include errors in the study’s methods or analysis that raise questions about the findings, or sections that need clearer explanations so that the manuscript is easily understood. From a journal editor’s point of view, comments on the importance and novelty of a manuscript, and if it will interest the journal’s audience, are particularly useful in helping them to decide which manuscripts to publish.

Will the authors know I am a reviewer? Will I know who the authors are? 

Traditionally, peer review worked in a way we now call “closed,” where the editor and the reviewers knew who the authors were, but the authors did not know who the reviewers were. In recent years, however, many journals have begun to develop other approaches to peer review. These include:

  • Closed peer review — where the reviewers are aware of the authors’ identities but the authors’ are never informed of the reviewers’ identities.
  • Double-blind peer review —where neither author nor reviewer is aware of each other’s identities.
  • Open peer review —where authors and reviewers are aware of each other’s identity. In some journals with open peer review the reviewers’ reports are published alongside the article.

The type of peer review used by a journal should be clearly stated in the invitation to review letter you receive and policy pages on the journal website. If, after checking the journal website, you are unsure of the type of peer review used or would like clarification on the journal’s policy you should contact the journal’s editors.

Why serve as a peer reviewer?

As your career advances, you are likely to be asked to serve as a peer reviewer.

As well as supporting the advancement of science, and providing guidance on how the author can improve their paper, there are also some benefits of peer reviewing to you as a researcher:

  • Serving as a peer reviewer looks good on your CV as it shows that your expertise is recognized by other scientists. (See the supplemental material about the Web of Science Reviewer Recognition Service to learn more about getting credit for the reviews you do. Also see the supplemental material about ORCiD iDs to learn how to connect your reviews to your unique ORCiD iD.) 
  • You will get to read some of the latest science in your field well before it is in the public domain.
  • The critical thinking skills needed during peer review will help you in your own research and writing.

Who does peer review benefit?

When performed correctly peer review helps improve the clarity, robustness and reproducibility of research.

When peer reviewing, it is helpful to think from the point of view of three different groups of people:

  • Authors . Try to review the manuscript as you would like others to review your work. When you point out problems in a manuscript, do so in a way that will help the authors to improve the manuscript. Even if you recommend to the editor that the manuscript be rejected, your suggested revisions could help the authors prepare the manuscript for submission to a different journal. 
  • Journal editors . Comment on the importance and novelty of the study. Editors will use your comments to assess whether the manuscript is of the right level of impact for the journal. Your comments and opinions on the paper are much more important that a simple recommendation; editors need to know why you think a paper should be published or rejected as your reasoning will help inform their decision.
  • Readers . Identify areas that need clarification to make sure other readers can easily understand the manuscript. As a reviewer, you can also save readers’ time and frustration by helping to keep unimportant or error filled research out of the published literature.

Writing a thorough, thoughtful review usually takes several hours or more. But by taking the time to be a good reviewer, you will be providing a service to the scientific community.  

Accepting an invitation to review

Editors invite you to review as they believe that you are an expert in a certain area. They would have judged this from your previous publication record or posters and/or sessions you have contributed to at conferences. You may find that the number of invitations to review increases as you progress in your career.

There are several questions to consider before you accept an invitation to review a paper.

  • Are you qualified? The editor has asked you to review the manuscript because he or she believes you are familiar with the specific topic or research method used in the paper. It will usually be okay if you can review some, but not all, aspects of a manuscript. Take as an example, if the study focused on a certain physiological process in an animal model you conduct your research on but used a technique that you have never used. In this case, simply review the parts of the manuscript that are in your area of expertise, and tell the editor which parts you cannot review. However, if the manuscript is too far outside your area, you should decline to review it.
  • Do you have time? If you know you will not be able to review the manuscript by the deadline, then you should not accept the invitation. Sending in a review long after the deadline will delay the publication process and frustrate the editor and authors. Keep in mind that reviewing manuscripts, like research and teaching, is a valuable contribution to science, and is worth making time for whenever possible.
  • The reported results could cause you to make or lose money, e.g., the authors are developing a drug that could compete with a drug you are working on.
  • The manuscript concerns a controversial question that you have strong feelings about (either agreeing or disagreeing with the authors).
  • You have strong positive or negative feelings about one of the authors, e.g., a former teacher who you admire greatly.
  • You have published papers or collaborated with one of the co-authors in recent years.

If you are not sure if you have a conflict of interest, discuss your circumstances with the editor.

Along with avoiding a conflict of interest, there are several other ethical guidelines to keep in mind as you review the manuscript. Manuscripts under review are highly confidential, so you should not discuss the manuscript – or even mention its existence – to others. One exception is if you would like to consult with a colleague about your review; in this case, you will need to ask the editor’s permission. It is normally okay to ask one of your students or postdocs to help with the review. However, you should let the editor know that you are being helped, and tell your assistant about the need for confidentiality. In some cases case, when the journal operates an open peer review policy they will allow the student or postdoc to co-sign the report with you should they wish.

It is very unethical to use information in the manuscript to make business decisions, such as buying or selling stock. Also, you should never plagiarize the content or ideas in the manuscript.

Next: Evaluating manuscripts

For further support

We hope that with this tutorial you have a clearer idea of how the peer review process works and feel confident in becoming a peer reviewer.

If you feel that you would like some further support with writing, reviewing, and publishing, Springer Nature offer some services which may be of help.

  • Nature Research Editing Service offers high quality  English language and scientific editing. During language editing , Editors will improve the English in your manuscript to ensure the meaning is clear and identify problems that require your review. With Scientific Editing experienced development editors will improve the scientific presentation of your research in your manuscript and cover letter, if supplied. They will also provide you with a report containing feedback on the most important issues identified during the edit, as well as journal recommendations.
  • Our affiliates American Journal Experts also provide English language editing* as well as other author services that may support you in preparing your manuscript.
  • We provide both online and face-to-face training for researchers on all aspects of the manuscript writing process.

* Please note, using an editing service is neither a requirement nor a guarantee of acceptance for publication. 

Test your knowledge

Take the Quiz!

Stay up to date

Here to foster information exchange with the library community

Connect with us on LinkedIn and stay up to date with news and development.

  • Tools & Services
  • Account Development
  • Sales and account contacts
  • Professional
  • Press office
  • Locations & Contact

We are a world leading research, educational and professional publisher. Visit our main website for more information.

  • © 2023 Springer Nature
  • General terms and conditions
  • Your US State Privacy Rights
  • Your Privacy Choices / Manage Cookies
  • Accessibility
  • Legal notice
  • Help us to improve this site, send feedback.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • What Is Peer Review? | Types & Examples

What Is Peer Review? | Types & Examples

Published on December 17, 2021 by Tegan George . Revised on June 22, 2023.

Peer review, sometimes referred to as refereeing , is the process of evaluating submissions to an academic journal. Using strict criteria, a panel of reviewers in the same subject area decides whether to accept each submission for publication.

Peer-reviewed articles are considered a highly credible source due to the stringent process they go through before publication.

There are various types of peer review. The main difference between them is to what extent the authors, reviewers, and editors know each other’s identities. The most common types are:

  • Single-blind review
  • Double-blind review
  • Triple-blind review

Collaborative review

Open review.

Relatedly, peer assessment is a process where your peers provide you with feedback on something you’ve written, based on a set of criteria or benchmarks from an instructor. They then give constructive feedback, compliments, or guidance to help you improve your draft.

Table of contents

What is the purpose of peer review, types of peer review, the peer review process, providing feedback to your peers, peer review example, advantages of peer review, criticisms of peer review, other interesting articles, frequently asked questions about peer reviews.

Many academic fields use peer review, largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the manuscript. For this reason, academic journals are among the most credible sources you can refer to.

However, peer review is also common in non-academic settings. The United Nations, the European Union, and many individual nations use peer review to evaluate grant applications. It is also widely used in medical and health-related fields as a teaching or quality-of-care measure.

Peer assessment is often used in the classroom as a pedagogical tool. Both receiving feedback and providing it are thought to enhance the learning process, helping students think critically and collaboratively.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Depending on the journal, there are several types of peer review.

Single-blind peer review

The most common type of peer review is single-blind (or single anonymized) review . Here, the names of the reviewers are not known by the author.

While this gives the reviewers the ability to give feedback without the possibility of interference from the author, there has been substantial criticism of this method in the last few years. Many argue that single-blind reviewing can lead to poaching or intellectual theft or that anonymized comments cause reviewers to be too harsh.

Double-blind peer review

In double-blind (or double anonymized) review , both the author and the reviewers are anonymous.

Arguments for double-blind review highlight that this mitigates any risk of prejudice on the side of the reviewer, while protecting the nature of the process. In theory, it also leads to manuscripts being published on merit rather than on the reputation of the author.

Triple-blind peer review

While triple-blind (or triple anonymized) review —where the identities of the author, reviewers, and editors are all anonymized—does exist, it is difficult to carry out in practice.

Proponents of adopting triple-blind review for journal submissions argue that it minimizes potential conflicts of interest and biases. However, ensuring anonymity is logistically challenging, and current editing software is not always able to fully anonymize everyone involved in the process.

In collaborative review , authors and reviewers interact with each other directly throughout the process. However, the identity of the reviewer is not known to the author. This gives all parties the opportunity to resolve any inconsistencies or contradictions in real time, and provides them a rich forum for discussion. It can mitigate the need for multiple rounds of editing and minimize back-and-forth.

Collaborative review can be time- and resource-intensive for the journal, however. For these collaborations to occur, there has to be a set system in place, often a technological platform, with staff monitoring and fixing any bugs or glitches.

Lastly, in open review , all parties know each other’s identities throughout the process. Often, open review can also include feedback from a larger audience, such as an online forum, or reviewer feedback included as part of the final published product.

While many argue that greater transparency prevents plagiarism or unnecessary harshness, there is also concern about the quality of future scholarship if reviewers feel they have to censor their comments.

In general, the peer review process includes the following steps:

  • First, the author submits the manuscript to the editor.
  • Reject the manuscript and send it back to the author, or
  • Send it onward to the selected peer reviewer(s)
  • Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made.
  • Lastly, the edited manuscript is sent back to the author. They input the edits and resubmit it to the editor for publication.

The peer review process

In an effort to be transparent, many journals are now disclosing who reviewed each article in the published product. There are also increasing opportunities for collaboration and feedback, with some journals allowing open communication between reviewers and authors.

It can seem daunting at first to conduct a peer review or peer assessment. If you’re not sure where to start, there are several best practices you can use.

Summarize the argument in your own words

Summarizing the main argument helps the author see how their argument is interpreted by readers, and gives you a jumping-off point for providing feedback. If you’re having trouble doing this, it’s a sign that the argument needs to be clearer, more concise, or worded differently.

If the author sees that you’ve interpreted their argument differently than they intended, they have an opportunity to address any misunderstandings when they get the manuscript back.

Separate your feedback into major and minor issues

It can be challenging to keep feedback organized. One strategy is to start out with any major issues and then flow into the more minor points. It’s often helpful to keep your feedback in a numbered list, so the author has concrete points to refer back to.

Major issues typically consist of any problems with the style, flow, or key points of the manuscript. Minor issues include spelling errors, citation errors, or other smaller, easy-to-apply feedback.

Tip: Try not to focus too much on the minor issues. If the manuscript has a lot of typos, consider making a note that the author should address spelling and grammar issues, rather than going through and fixing each one.

The best feedback you can provide is anything that helps them strengthen their argument or resolve major stylistic issues.

Give the type of feedback that you would like to receive

No one likes being criticized, and it can be difficult to give honest feedback without sounding overly harsh or critical. One strategy you can use here is the “compliment sandwich,” where you “sandwich” your constructive criticism between two compliments.

Be sure you are giving concrete, actionable feedback that will help the author submit a successful final draft. While you shouldn’t tell them exactly what they should do, your feedback should help them resolve any issues they may have overlooked.

As a rule of thumb, your feedback should be:

  • Easy to understand
  • Constructive

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

  • Academic style
  • Vague sentences
  • Style consistency

See an example

peer review 03

Below is a brief annotated research example. You can view examples of peer feedback by hovering over the highlighted sections.

Influence of phone use on sleep

Studies show that teens from the US are getting less sleep than they were a decade ago (Johnson, 2019) . On average, teens only slept for 6 hours a night in 2021, compared to 8 hours a night in 2011. Johnson mentions several potential causes, such as increased anxiety, changed diets, and increased phone use.

The current study focuses on the effect phone use before bedtime has on the number of hours of sleep teens are getting.

For this study, a sample of 300 teens was recruited using social media, such as Facebook, Instagram, and Snapchat. The first week, all teens were allowed to use their phone the way they normally would, in order to obtain a baseline.

The sample was then divided into 3 groups:

  • Group 1 was not allowed to use their phone before bedtime.
  • Group 2 used their phone for 1 hour before bedtime.
  • Group 3 used their phone for 3 hours before bedtime.

All participants were asked to go to sleep around 10 p.m. to control for variation in bedtime . In the morning, their Fitbit showed the number of hours they’d slept. They kept track of these numbers themselves for 1 week.

Two independent t tests were used in order to compare Group 1 and Group 2, and Group 1 and Group 3. The first t test showed no significant difference ( p > .05) between the number of hours for Group 1 ( M = 7.8, SD = 0.6) and Group 2 ( M = 7.0, SD = 0.8). The second t test showed a significant difference ( p < .01) between the average difference for Group 1 ( M = 7.8, SD = 0.6) and Group 3 ( M = 6.1, SD = 1.5).

This shows that teens sleep fewer hours a night if they use their phone for over an hour before bedtime, compared to teens who use their phone for 0 to 1 hours.

Peer review is an established and hallowed process in academia, dating back hundreds of years. It provides various fields of study with metrics, expectations, and guidance to ensure published work is consistent with predetermined standards.

  • Protects the quality of published research

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. Any content that raises red flags for reviewers can be closely examined in the review stage, preventing plagiarized or duplicated research from being published.

  • Gives you access to feedback from experts in your field

Peer review represents an excellent opportunity to get feedback from renowned experts in your field and to improve your writing through their feedback and guidance. Experts with knowledge about your subject matter can give you feedback on both style and content, and they may also suggest avenues for further research that you hadn’t yet considered.

  • Helps you identify any weaknesses in your argument

Peer review acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process. This way, you’ll end up with a more robust, more cohesive article.

While peer review is a widely accepted metric for credibility, it’s not without its drawbacks.

  • Reviewer bias

The more transparent double-blind system is not yet very common, which can lead to bias in reviewing. A common criticism is that an excellent paper by a new researcher may be declined, while an objectively lower-quality submission by an established researcher would be accepted.

  • Delays in publication

The thoroughness of the peer review process can lead to significant delays in publishing time. Research that was current at the time of submission may not be as current by the time it’s published. There is also high risk of publication bias , where journals are more likely to publish studies with positive findings than studies with negative findings.

  • Risk of human error

By its very nature, peer review carries a risk of human error. In particular, falsification often cannot be detected, given that reviewers would have to replicate entire experiments to ensure the validity of results.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

  • Normal distribution
  • Measures of central tendency
  • Chi square tests
  • Confidence interval
  • Quartiles & Quantiles
  • Cluster sampling
  • Stratified sampling
  • Thematic analysis
  • Discourse analysis
  • Cohort study
  • Ethnography

Research bias

  • Implicit bias
  • Cognitive bias
  • Conformity bias
  • Hawthorne effect
  • Availability heuristic
  • Attrition bias
  • Social desirability bias

Peer review is a process of evaluating submissions to an academic journal. Utilizing rigorous criteria, a panel of reviewers in the same subject area decide whether to accept each submission for publication. For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

In general, the peer review process follows the following steps: 

  • Reject the manuscript and send it back to author, or 
  • Send it onward to the selected peer reviewer(s) 
  • Next, the peer review process occurs. The reviewer provides feedback, addressing any major or minor issues with the manuscript, and gives their advice regarding what edits should be made. 
  • Lastly, the edited manuscript is sent back to the author. They input the edits, and resubmit it to the editor for publication.

Peer review can stop obviously problematic, falsified, or otherwise untrustworthy research from being published. It also represents an excellent opportunity to get feedback from renowned experts in your field. It acts as a first defense, helping you ensure your argument is clear and that there are no gaps, vague terms, or unanswered questions for readers who weren’t involved in the research process.

Peer-reviewed articles are considered a highly credible source due to this stringent process they go through before publication.

Many academic fields use peer review , largely to determine whether a manuscript is suitable for publication. Peer review enhances the credibility of the published manuscript.

However, peer review is also common in non-academic settings. The United Nations, the European Union, and many individual nations use peer review to evaluate grant applications. It is also widely used in medical and health-related fields as a teaching or quality-of-care measure. 

A credible source should pass the CRAAP test  and follow these guidelines:

  • The information should be up to date and current.
  • The author and publication should be a trusted authority on the subject you are researching.
  • The sources the author cited should be easy to find, clear, and unbiased.
  • For a web source, the URL and layout should signify that it is trustworthy.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). What Is Peer Review? | Types & Examples. Scribbr. Retrieved February 26, 2024, from https://www.scribbr.com/methodology/peer-review/

Is this article helpful?

Tegan George

Tegan George

Other students also liked, what are credible sources & how to spot them | examples, ethical considerations in research | types & examples, applying the craap test & evaluating sources, what is your plagiarism score.

Library homepage

  • school Campus Bookshelves
  • menu_book Bookshelves
  • perm_media Learning Objects
  • login Login
  • how_to_reg Request Instructor Account
  • hub Instructor Commons
  • Download Page (PDF)
  • Download Full Book (PDF)
  • Periodic Table
  • Physics Constants
  • Scientific Calculator
  • Reference & Cite
  • Tools expand_more
  • Readability

selected template will load here

This action is not available.

Humanities LibreTexts

5.3: Peer Review as Collaboration

  • Last updated
  • Save as PDF
  • Page ID 6491

  • Steven D. Krause
  • Eastern Michigan University

One of the most common types of collaboration done in writing classes comes in the form of in-class “group work” or as peer review sessions. Peer review has become a common practice for contemporary composition and rhetoric classrooms. Basically, it is the process where small groups of students read, comment on, and make suggestions for other student’s work.

While successful peer review can be hard and takes practice, it really can work. But first, you have to be willing to accept two premises.

  • Your fellow students have valid comments to make on your writing projects. Students often assume that the only person whose opinion really matters is the teacher because, after all, the teacher is the one who assigns the grade. I understand the logic of the assumption that the “teacher is always right,” but I don’t think it’s true.

The best writing projects are ones that strive to fulfill a purpose and reach an audience that is beyond a particular class and a particular teacher. But beyond that, your classmates represent an audience you should be trying to reach. You should listen to your classmate’s suggestions because they are in same writing situation as you. After all, they too are trying to reach an audience that includes their fellow classmates, and they are also writing a project that will have to be read and evaluated by the teacher.

  • All writing projects can be improved by revision. Sometimes we have an overly romantic view of writing and of writers who are able to create “great works” without ever having to make any real changes. Rarely (if ever) has this been the case. Any writing project can be improved with revision.

As straightforward as these premises might be, they can often be difficult to accept. But with practice, patience, and work with your classmates, seeing these premises as valid becomes easier.

How peer review can work, step by step

I offer the following advice on how to get started with peer review sessions as a “recipe” where ingredients and methods can be altered to fit the particulars of the class, the writing project, time limitations, and so forth. After all, you and your teacher probably have ideas on what will or won’t work for peer review in your specific contexts.

  • With the help of your teacher, break into groups of three to five students. Groups of five work well only if the writing project you are considering is short or if you have a lot of class time to go over each project. I would also recommend not working in pairs since that overly limits the size of the audience.

Some students and teachers like to work with the same peer collaborators for the entire semester, while others like to work with different collaborators with each project.

  • Exchange a copy of your writing project with each person in the group. You should come to the peer review session class with several copies of your writing project to share with others in your peer review group.
  • Select someone to start, and have that person read their essay out loud while the other members of the group read along. The extent to which you will be able to read your essays out loud will vary according to the particular circumstances of your class and of the assignment, but I would encourage you to try to include this step in the process of in-class peer review. Actually reading your writing out loud to others gives the reader and writer a real sense of the voice of an essay and is a great way for writers and readers to catch small grammar errors.
  • While the writer “up” is reading, the readers should read along, marking comments in the margins of the draft they are reading. As a reader, you should note points you hope to come back to in group discussion. You can also mark any grammatical errors you might notice as you read.
  • When the writer is done reading, the readers should provide their comments. This is not the time for the writer to explain things that the readers say they didn’t understand. Rather, this is the time for the writer to listen to what the other members of the group have to say.

This is a crucial part of the process because the questions that readers have are ones that point to changes the writer should make in revision rather than being answered in person. After all, you will never be able to be there when other readers (your teacher or other people in your audience) try to understand your writing project. Readers’ questions have to be anticipated and answered in the writing itself. So, the role of the person who just finished reading is to try and be as open-minded (and open-eared!) to their classmates’ advice as possible.

Giving good advice to classmates in peer review sessions can be a tricky process. Readers often have a hard time expressing their comments to the person who’s writing is being discussed. On the one hand, it isn’t productive or nice to say things that might hurt the writer’s feelings; but on the other hand, it also isn’t productive to be so nice as to not say anything that can help the writer. So the goal here should be to somehow balance the two: advice that is “nice,” but also constructive.

Here are two suggestions to help make this step of readers giving writers constructive advice a bit easier:

  • Try to keep the focus of the constructive advice on the big issues. By “the big issues,” I mean things like the clarity of the points the writer is trying to make, the use of evidence, the points where readers are particularly persuaded or particularly confused, and so forth. This is not to say things like grammar and proofreading and such are not important—far from it. But those issues are more about “proofreading” than they are about changing the substance of an essay.
  • Consider some of the questions I have at the end of each of the chapters in Part Two of the book, “Exercises in the Process of Research .” Each of the chapters in this part of the book end with sections titled “Questions to Ask While Writing and Researching” and “Review and Revision.” The questions you should consider very according to the writing exercise, but the goal is always the same: what changes can you make to your writing project to make it more accessible to your readers?

Making revisions as a result questions like these (and the ones provided by your teacher) will make it much easier for you and your group members to give each other useful advice, and it will also help keep the group on task.

A few final things to remember about successful peer review

  • Peer review takes practice. If you don’t think peer review works that well for you and your classmates the first time you try it, give it another chance with a different writing project. Like most things in writing (or life!) that are rewarding and useful, good peer review takes practice and time. If you stick with it, you’ll see that the peer review sessions you have toward the end of term are much more productive than the ones at the beginning of the term.
  • If you don’t get good advice about your writing projects in class, seek out advice elsewhere. Show a draft of your writing project to someone who’s opinion you value—friends, family, classmates—and ask them for suggestions in making the project better. If your school has a writing center, writing lab, or other sort of tutoring center, take a copy of the writing project to it and have a staff member look at your work.
  • It is always still up to you to choose what advice you want to follow. Inevitably, you will receive advice from your reviewers that is conflicting or that is advice you simply don’t agree with. That is okay. Remember that you are not under any obligation to incorporate all the suggestions you receive, and part of the process of becoming a better writer is learning for yourself when you need to follow advice and when you need to follow your own instincts.

A middle aged man sits at a computer against a wall full of books.

Peer review isn’t perfect − I know because I teach others how to do it and I’ve seen firsthand how it comes up short

peer review 03

Director of the Center for Teaching and Learning, Quinnipiac University

Disclosure statement

JT Torres does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

Quinnipiac University provides funding as a member of The Conversation US.

View all partners

When I teach research methods, a major focus is peer review . As a process, peer review evaluates academic papers for their quality, integrity and impact on a field, largely shaping what scientists accept as “knowledge.” By instinct, any academic follows up a new idea with the question, “Was that peer reviewed?”

Although I believe in the importance of peer review – and I help do peer reviews for several academic journals – I know how vulnerable the process can be. Not only have academics questioned peer review reliability for decades, but the retraction of more than 10,000 research papers in 2023 set a new record.

I had my first encounter with the flaws in the peer review process in 2015, during my first year as a Ph.D. student in educational psychology at a large land-grant university in the Pacific Northwest.

My adviser published some of the most widely cited studies in educational research. He served on several editorial boards. Some of the most recognized journals in learning science solicited his review of new studies. One day, I knocked on his office door. He answered without getting up from his chair, a printed manuscript splayed open on his lap, and waved me in.

“Good timing,” he said. “Do you have peer review experience?”

I had served on the editorial staff for literary journals and reviewed poetry and fiction submissions, but I doubted much of that transferred to scientific peer review.

“Fantastic.” He smiled in relief. “This will be real-world learning.” He handed me the manuscript from his lap and told me to have my written review back to him in a week.

I was too embarrassed to ask how one actually does peer review, so I offered an impromptu plan based on my prior experience: “I’ll make editing comments in the margins and then write a summary about the overall quality?”

His smile faded, either because of disappointment or distraction. He began responding to an email.

“Make sure the methods are sound. The results make sense. Don’t worry about the editing.”

Ultimately, I fumbled my way through, saving my adviser time on one less review he had to conduct. Afterward, I did receive good feedback and eventually became a confident peer reviewer. But at the time, I certainly was not a “peer.” I was too new in my field to evaluate methods and results, and I had not yet been exposed to enough studies to identify a surprising observation or to recognize the quality I was supposed to control. Manipulated data or subpar methods could easily have gone undetected.

Effects of bias

Knowledge is not self-evident. A survey can be designed with a problematic amount of bias , even if unintentional.

Observing a phenomenon in one context, such as an intervention helping white middle-class children learn to read, may not necessarily yield insights for how to best teach reading to children in other demographics. Debates over “the science of reading” in general have lasted decades, with researchers arguing over constantly changing “recommendations ,” such as whether to teach phonics or the use of context cues.

A correlation – a student who bullies other students and plays violent video games – may not be causation . We do not know if the student became a bully because of playing violent video games. Only experts within a field would be able to notice such differences, and even then, experts do not always agree on what they notice.

Four researchers look at an open notebook.

As individuals, we can very often be limited by our own experiences. Let’s say in my life I only see white swans. I might form the knowledge that only white swans exist. Maybe I write a manuscript about my lifetime of observations, concluding that all swans are white. I submit that manuscript to a journal, and a “peer,” someone who also has observed a lot of swans, says, “Wait a minute, I’ve seen black swans.” That peer would communicate back to me their observations so that I can refine my knowledge.

The peer plays a pivotal role evaluating observations, with the overall goal of advancing knowledge. For example, if the above scenario were reversed, and peer reviewers who all believed that all swans were white came across the first study observing a black swan, the study would receive a lot of attention as researchers scrambled to replicate that observation. So why was a first-year graduate student getting to stand in for an expert? Why would my review count the same as a veteran’s review? One answer: The process relies almost entirely on unpaid labor .

Despite the fact that peers are professionals, peer review is not a profession.

As a result, the same overworked scholars often receive the bulk of the peer review requests. Besides the labor inequity, a small pool of experts can lead to a narrowed process of what is publishable or what counts as knowledge, directly threatening diversity of perspectives and scholars .

Without a large enough reviewer pool, the process can easily fall victim to politics, arising from a small community recognizing each other’s work and compromising conflicts of interest. Many of the issues with peer review can be addressed by professionalizing the field, either through official recognition or compensation.

Value despite challenges

Despite these challenges, I still tell my students that peer review offers the best method for evaluating studies and advancing knowledge. Consider the statistical phenomenon suggesting that groups of people are more likely to arrive at “right answers” than individuals.

In his book “ The Wisdom of Crowds ,” author James Surowiecki tells the story of a county fair in 1906, where fairgoers guessed the weight of an ox. Sir Francis Galton averaged the 787 guesses and arrived at 1,197 pounds. The ox weighed 1,198 pounds.

When it comes to science and the reproduction of ideas, the wisdom of the many can account for individual outliers. Fortunately, and ironically, this is how science discredited Galton’s take on eugenics, which has overshadowed his contributions to science .

As a process, peer review theoretically works. The question is whether the peer will get the support needed to effectively conduct the review.

  • Peer review
  • Retractions
  • Academic journal
  • Scholarship
  • Higher ed attainment

peer review 03

Research Assistant in Immunology and Virology

peer review 03

General Manager | La Trobe University, Sydney Campus

peer review 03

Lecturer / Senior Lecturer - Business Law & Taxation

peer review 03

Industrial Officer (Senior)

peer review 03

Supply Chain Management – Open Rank (Tenure-Track)

The Royal Society

The future of peer review in the 17th Century

Transparency and equity have become central values in modern academic publishing—but how did we get here.

Max Born’s report on 'The accuracy of atomic co-ordinates derived from Fourier series in X-ray structure analysis'. Science in the Making.

For scientists in the 17th Century, what did the future of peer review and the publishing process look like?  Philosophical Transactions of the Royal Society , the world’s first and longest-running scientific journal, was launched in 1665. However, the scientific community would have to wait almost two hundred years before a formalised peer review process was introduced. 

Things can only get better

For almost a hundred years , editorial power lay solely with the journal Editor, beginning with the founding Editor and the Royal Society’s first Secretary, Henry Oldenburg FRS. With one man serving as gatekeeper to publication in the journal, there was much room for progress.

By the 1750s though, a change was deemed warranted by the Society , resulting in the dilution of editorial power to a committee of Fellows. The movement towards democratic decision making, and the inclusion of different expertise and perspectives, reflects a more modern approach that would help lay the foundations for the peer review and editorial board structures of today. To meet the demand of a growing and diversifying scientific output in the years that followed, progress was made again in 1832 in the form of a nascent, formalised peer review process that utilised the expertise of the wider Fellowship.

Science in the Making – the newly digitised trove of the Royal Society’s archives – provides a window into the past of scholarly publishing practices, including these first steps of peer review. The insightful collection of referee reports sheds light on the evolving peer review process at the Royal Society from 1832-1949.

Open and shut

Taking inspiration from other European science academies at the time, the Royal Society’s editorial committee made an early endeavour into open peer review in 1831-32. In the same period that saw the introduction of a more formalised and thorough reviewing process, the Royal Society trialled the publication of reports in the newly established Proceedings journal, a companion to the Philosophical Transactions where the corresponding paper would appear.

Title page from the joint referees’ report by Samuel Hunter Christie and John Bostock, on a paper 'Experimental researches in electricity' by Michael Faraday.

'Report on Faraday paper, April 5th 1832'. Title page from the joint referees’ report by Samuel Hunter Christie and John Bostock, which was made available alongside the accepted paper, ' Experimental researches in electricity ' by Michael Faraday. Both referees agreed on the paper’s importance.

The plan was for sets of referees to produce joint reports that would be made available with the accepted paper. The collaborative nature of the trial proved a struggle though—where some agreed strongly (as in the case of the Faraday paper above), others, naturally, experienced some disagreement. 

In combination with concerns over confidentiality and personal reputations, the additional time and effort to prepare these reports collaboratively, and in a fashion that was fit for publication, deterred the editorial committee from continuing this open practice.

The Royal Society would have to wait until the 21st Century before revisiting open peer review, where it is now an established component of publishing, and a cornerstone of the open science ethos more generally. A key difference between then and now though is the independence of referees.

Independence and bias

Though the idea of seeking two experts two assess your manuscript may have seemed like progress, and more favourable to an author than the ruling of a single editor or a committee vote, there was of course still room for bias. There were often instances of referee 1 forwarding on the manuscript, plus their own report, on to a second referee, implying that one opinion might not be totally independent of the other. Independence is of course valued more highly today, and much more easily managed with online systems (rather than the postal service).

Not that scientists were unaware of the qualities of bias, as illustrated by Charles Darwin’s 1879 referee report on John Prestwich’s Scottish highlands geology paper, which includes a declaration of personal interest and a suggestion that he may be inclined to overestimate the usefulness and value of the paper—although, only as a final point. 

Extract from Darwin’s report on 'On the origin of the parallel roads of Lochaber and their bearing on other phenomena of the glacial period'.

Extract from Darwin’s report on 'On the origin of the parallel roads of Lochaber and their bearing on other phenomena of the glacial period'. Mercifully clear handwriting compared to some contemporaries. Typewriters and word processors would banish ink blots and impenetrable scroll.

Same old excuse

Amongst the perennial reasons for declined reviewer invitations and delayed responses—namely busy academic and personal lives—there were also era-specific causes, as alluded to in a 1942 review of a matrix theory paper, where two recommended referees were ‘busy with war work.’ The credibility of these excuses varied though—in 1926, mathematician Louis N. G. Filon FRS simply had an unavoidable yachting holiday.

Identifying the right candidate to review a paper appears to have always proved a challenge, with numerous examples of invited referees returning short, apologetic reports recommending alternative expertise be sought. Even Nobel Laureate Max Born found himself assigned papers beyond his abilities, such as an X-ray crystallography paper that he recommends be assessed by alternatives such as Lawrence Bragg and Kathleen Lonsdale.

Max Born’s brief report on 'The accuracy of atomic co-ordinates derived from Fourier series in X-ray structure analysis'

Max Born’s brief report on 'The accuracy of atomic co-ordinates derived from Fourier series in X-ray structure analysis'. By the mid-20th century the reports had become more standardised with set questions concerning suitability, redundancy and interest, reflecting modern peer review reports. Question 7 is a reminder of the era though.  

Experience shouldn’t be everything though. In 1875, the astronomer/inventor Warren De la Rue wholeheartedly recommended for publication the somewhat niche paper 'On polishing the specula of reflecting telescopes' by William Lassell, declaring that 'no-one has had more experience in the production of specula than Mr Lassell'. Thankfully, his report appears to be detailed.

Explore all referee reports, published papers and more with Science in the Making . For more on the history of publishing at the Royal Society, see A History of Scientific Journals Publishing at the Royal Society, 1665–2015 .

Callum Shoosmith

Callum Shoosmith

Editorial Coordinator, the Royal Society

Related blogs

peer review 03

Buchi Okereafor

In publishing

AI and the future of scholarly publishing (part 1)

The focus for this year's Peer Review Week is 'Peer Review and the Future of Publishing'. We speak…

peer review 03

AI and the future of scholarly publishing (part 2)

Matt Hodgkinson, Research Integrity Manager at the UK Research Integrity Office, gives us an…

peer review 03

Andrew Dunn

Peer Review and The Future of Publishing

The focus for this year's Peer Review Week is 'Peer Review and the Future of Publishing'. In this…

Skip Navigation

  • Scientific Research
  • Professional Development
  • Career Paths

Image of three blue squares stacked vertically to look like pages.

  • About Neuronline
  • Community Leaders Program
  • Write for Us
  • Community Guidelines

Neuronline logo

  • COLLECTIONS

How to Navigate the Peer Review Process

  • Event Policies
  • Stay Connected

peer review 03

You’ve chosen your journal and submitted your paper, but what happens next? Join this hour-long webinar to hear Editors-in-Chief Sabine Kastner ( JNeurosci ) and Mariela Zirlinger ( Neuron ) discuss how editors and reviewers are assigned, desk rejections, and responding to reviewer feedback. Attendees are encouraged to submit questions at registration and live during the webinar.

This webinar is part of a series from SfN Journals and Elsevier covering topics related to the publishing process.

Image of Sabine Kastner

Who can attend these webinars? This webinar is open access and registration is complimentary.

Will the webinars be available on-demand? Yes, this webinar will be available on demand 1 week from the live broadcast .

How do I access the conference on the live day? After registering, you will receive a confirmation email with the event link and the option to download calendar reminders.

What are the technology requirements for attending?  These webinars are hosted on Zoom Webinar. Instructions for joining and participating in a webinar can be found  here .

Can I ask the presenters questions?  Yes! You can submit any questions before the webinar through the registration form. During the webinar, you can submit questions through the Q&A box.

Will a certificate of attendance be offered for this event? No, SfN does not provide certificates of attendance for webinars. 

I have other questions not answered here. Email [email protected]  with any other questions. 

Review SfN’s Code of Conduct , rules for virtual events in the Digital Learning Community Guidelines , and communications policies regarding dissemination of unpublished scientific data, listed below. SfN asks that conference attendees respect the sensitivity of information and data being presented that are not yet available to the public by following these guidelines:

  • Do not capture or publicly share details of any unpublished data presented.
  • If you are unsure whether data is unpublished, check with the presenter.
  • Respect presenters' wishes if they indicate that the information presented is not to be shared.

More in Scientific Research

Data and coding. Photo by Bibek ghosh

  • Accessibility Policy
  • Privacy Policy
  • Manage Cookies

SfN logo with "SfN" in a blue box next to Society for Neuroscience in red text and the SfN tag line that reads "Advancing the understanding of the brain and nervous system"

Copyright © 2019 Society for Neuroscience

  • Mobile Site
  • Staff Directory
  • Advertise with Ars

Filter by topic

  • Biz & IT
  • Gaming & Culture

Front page layout

AI gone wild —

Scientists aghast at bizarre ai rat with huge genitals in peer-reviewed article, it's unclear how such egregiously bad images made it through peer-review..

Beth Mole - Feb 15, 2024 11:16 pm UTC

An actual laboratory rat, who is intrigued.

Appall and scorn ripped through scientists' social media networks Thursday as several egregiously bad AI-generated figures circulated from a peer-reviewed article recently published in a reputable journal. Those figures—which the authors acknowledge in the article's text were made by Midjourney—are all uninterpretable. They contain gibberish text and, most strikingly, one includes an image of a rat with grotesquely large and bizarre genitals, as well as a text label of "dck."

AI-generated Figure 1 of the paper. This image is supposed to show spermatogonial stem cells isolated, purified, and cultured from rat testes.

The article in question is titled "Cellular functions of spermatogonial stem cells in relation to JAK/STAT signaling pathway," which was authored by three researchers in China, including the corresponding author Dingjun Hao of Xi’an Honghui Hospital. It was published online Tuesday in the journal Frontiers in Cell and Developmental Biology.

Frontiers did not immediately respond to Ars' request for comment, but we will update this post with any response.

Figure 2 is supposed to be a diagram of the JAK-STAT signaling pathway.

But the rat's package is far from the only problem. Figure 2 is less graphic but equally mangled. While it's intended to be a diagram of a complex signaling pathway, it instead is a jumbled mess. One scientific integrity expert questioned whether it provided an overly complicated explanation of "how to make a donut with colorful sprinkles." Like the first image, the diagram is rife with nonsense text and baffling images. Figure 3 is no better, offering a collage of small circular images that are densely annotated with gibberish. The image is supposed to provide visual representations of how the signaling pathway from Figure 2 regulates the biological properties of spermatogonial stem cells.

Some scientists online questioned whether the article's text was also AI-generated. One user noted that AI detection software determined that it was likely to be AI-generated; however, as Ars has reported previously, such software is unreliable .

Figure 3 is supposed to show the regulation of biological properties of spermatogonial stem cells by JAK/STAT signaling pathway.

The images, while egregious examples, highlight a growing problem in scientific publishing. A scientist's success relies heavily on their publication record, with a large volume of publications, frequent publishing, and articles appearing in top-tier journals, all of which earn scientists more prestige. The system incentivizes less-than-scrupulous researchers to push through low-quality articles, which, in the era of AI chatbots, could potentially be generated with the help of AI. Researchers worry that the growing use of AI will make published research less trustworthy. As such, research journals have recently set new authorship guidelines for AI-generated text to try to address the problem. But for now, as the Frontiers article shows, there are clearly some gaps.

reader comments

Channel ars technica.

FactCheck.org

Review Article By Misinformation Spreaders Misleads About mRNA COVID-19 Vaccines

Scicheck digest.

T he mRNA COVID-19 vaccines have a good safety record and have saved millions of lives. But viral posts claim the contrary, citing a recent peer-reviewed article authored by known COVID-19 misinformation spreaders and published in a controversial journal. The paper repeats previously debunked claims.

More than  half a billion doses of COVID-19 vaccines have now been administered in the U.S. and only a few, very rare, safety concerns have emerged. The vast majority of people experience only minor, temporary side effects such as pain at the injection site, fatigue, headache, or muscle pain — or no side effects at all. As the Centers for Disease Control and Prevention has said , these vaccines “have undergone and will continue to undergo the most intensive safety monitoring in U.S. history.”

A small number of severe allergic reactions known as anaphylaxis, which are expected with any vaccine, have occurred with the authorized and approved COVID-19 vaccines. Fortunately, these reactions are rare, typically occur within minutes of inoculation and can be treated. Approximately 5 per million people vaccinated have experienced anaphylaxis after a COVID-19 vaccine, according  to the CDC.

To make sure serious allergic reactions can be identified and treated, all people receiving a vaccine should be observed for 15 minutes after getting a shot, and anyone who has experienced anaphylaxis or had any kind of immediate allergic reaction to any vaccine or injection in the past should be monitored for a half hour. People who have had a serious allergic reaction to a previous dose or one of the vaccine ingredients should not be immunized. Also, those who shouldn’t receive one type of COVID-19 vaccine should be monitored for 30 minutes after receiving a different type of vaccine.

There is evidence that the Pfizer/BioNTech and Moderna mRNA vaccines may rarely cause inflammation of the heart muscle (myocarditis) or of the surrounding lining (pericarditis), particularly in male adolescents and young adults .

Based on data collected through August 2021, the reporting rates of either condition in the U.S. are highest in males 16 to 17 years old after the second dose (105.9 cases per million doses of the Pfizer/BioNTech vaccine), followed by 12- to 15-year-old males (70.7 cases per million). The rate for 18- to 24-year-old males was 52.4 cases and 56.3 cases per million doses of Pfizer/BioNTech and Moderna vaccines, respectively.

Health officials have emphasized that vaccine-related myocarditis and pericarditis cases are rare and the benefits of vaccination still outweigh the risks. Early evidence suggests these myocarditis cases are less severe than typical ones. The CDC has also noted that most patients who were treated “responded well to medicine and rest and felt better quickly.”

The Johnson & Johnson vaccine has been linked to an  increased risk of rare blood clots combined with low levels of blood platelets, especially in women ages 30 to 49 . Early symptoms of the condition, which is known as thrombosis with thrombocytopenia syndrome, or TTS, can appear as late as three weeks after vaccination and  include  severe or persistent headaches or blurred vision, leg swelling, and easy bruising or tiny blood spots under the skin outside of the injection site.

According to the CDC, TTS has occurred in around 4 people per million doses administered. As of early April ,  the syndrome has been confirmed in 60 cases, including nine deaths, after more than 18.6 million doses of the J&J vaccine. Although TTS remains rare, because of the availability of mRNA vaccines, which are not associated with this serious side effect, the FDA on May 5 limited authorized use of the J&J vaccine to adults who either couldn’t get one of the other authorized or approved COVID-19 vaccines because of medical or access reasons, or only wanted a J&J vaccine for protection against the disease. Several months earlier, on Dec. 16, 2021 ,  the CDC had recommended the Pfizer/BioNTech and Moderna shots over J&J’s.

The J&J vaccine has also been linked to an increased risk of Guillain-Barré Syndrome, a rare disorder in which the immune system attacks nerve cells.  Most people  who develop GBS fully recover, although some have permanent nerve damage and the condition can be fatal.

Safety surveillance data suggest that compared with the mRNA vaccines, which have not been linked to GBS, the J&J vaccine is associated with 15.5 additional GBS cases per million doses of vaccine in the three weeks following vaccination. Most reported cases following J&J vaccination have occurred in men 50 years old and older.

Link to this

The  safety  of the mRNA COVID-19 vaccines from Pfizer/BioNTech and Moderna is supported by the rigorous clinical trials run prior to their release and numerous studies conducted since. Hundreds of millions of people have been vaccinated in the U.S., many with multiple doses, and serious side effects are rare .

COVID-19 vaccines have also been shown to be  effective  in reducing the risk of severe forms of the disease. Multiple studies have estimated that the COVID-19 vaccines saved millions of lives across the globe.

But an  article  — written by misinformation spreaders who oppose COVID-19 vaccination — that claims to have reviewed the original trials and “other relevant studies” largely ignores this body of evidence. Instead, the review, which calls for a “global moratorium” on the mRNA vaccines, cites multiple flawed or criticized studies —  many   of   which   we’ve   written about before — to falsely claim the mRNA COVID-19 vaccines have caused “extensive, well-documented” serious adverse events and have killed nearly 14 times as many people as they saved.

The article was peer-reviewed and published in Cureus, an open-access online medical journal that prioritizes fast publication and has published problematic studies before, as we will explain.

Social   media   posts  that share the incorrect conclusions of the review have gone viral. 

“mRNA COVID-19 vaccines caused more deaths than saved: study,” reads a Feb. 4  Instagram post  that shared a screenshot of a headline by the Epoch Times. 

One author of the review — as well as other social media users — are also using the fact that the paper was published as proof that the mRNA COVID-19 vaccines are unsafe.

“People have said I’m a misinformation spreader because since  May 2021, I have been publicly saying the COVID vaccines are not safe . Now the medical peer-reviewed literature shows I was right.  Do you believe me now? ” Steve Kirsch, a review co-author and a former tech entrepreneur who lacks biomedical training, said in a post on X, formerly known as Twitter, on Jan. 30 (emphasis is his). 

“!! TRUST THE #SCIENCE !!,” the author of a  viral post  wrote on Instagram on Feb. 7. The post included a screenshot of a news story titled “Mainstream science mulls ‘global moratorium’ on COVID vaccines as cancers rise, boosters flub,” and the statement “Covid vaccines *may* cause cancer. You don’t say.” 

Just because a paper is published does not make it correct. While peer review is useful in weeding out bad science, it’s not foolproof, and the rigor and processes vary by journal. This review, which many experts have criticized, is an outlier, not “mainstream science.” And as  we’ve   written , there’s no evidence mRNA COVID-19 vaccines cause cancer and resulted in millions of deaths. 

Anti-Vaccine Authors and Debunked Claims

Many of the review’s authors have a history of spreading COVID-19 or vaccine misinformation. This includes Kirsch , who has repeatedly pushed the incorrect idea that the COVID-19 vaccines have killed millions of people worldwide, as well as Dr. Peter McCullough , Stephanie Seneff and Jessica Rose.

McCullough still   recommends  treating COVID-19 patients with hydroxychloroquine and ivermectin, even though both have been shown not to work against the disease. He also promotes and sells “spike protein detoxification” products for people who have been vaccinated, despite no evidence that vaccinated people need any such detox.

Seneff is a computer scientist who has promoted the false notion that vaccines cause autism. She previously co-authored a review paper with McCullough, which the Cureus review cites, that misused data from the Vaccine Adverse Event Reporting System to baselessly claim the mRNA COVID-19 vaccines suppress the innate immune system, as we  reported . Rose has also been  accused  of misusing VAERS data to claim vaccines are not safe — a common deception among the anti-vaccination community.

The Cureus review cites and even republishes a figure from one of Rose’s Substack posts about the supposedly alarming number of VAERS reports for “autoimmune disorders” following COVID-19 vaccination compared with influenza vaccines. The review claims the increased reporting “represents an immense safety signal.” But as we’ve explained   before , the higher number of VAERS reports for the COVID-19 vaccines can be explained by multiple factors, such as increased awareness and stricter reporting requirements – and does not in and of itself constitute a safety signal. A report can be submitted by anyone and does not mean that a vaccine caused a particular problem.

The review paper, titled “COVID-19 mRNA Vaccines: Lessons Learned from the Registrational Trials and Global Vaccination Campaign,” repeats many claims we’ve already written about, based on studies or analyses that have been widely criticized or debunked. 

To claim the vaccines cause “serious harms to humans,” for example, the review draws on a problematic reanalysis of the adverse events reported in the original trials that was published in the journal Vaccine in 2022. Florida Gov. Ron DeSantis and Dr. Joseph Ladapo, the state’s surgeon general, have cited the paper to argue that the vaccines are too risky. But as  we’ve   written — and is detailed in a commentary article published in the same journal — the paper has multiple methodological flaws, including how it counted the adverse events.

The review also uncritically cites an unpublished analysis by former physics professor Denis Rancourt that alleged that some 17 million people died from the COVID-19 vaccines. We recently explained that the report erroneously ignored deaths from COVID-19 and that such estimates are implausible. And the review recycles unsupported claims about “high levels of DNA contamination” in the mRNA vaccines and the possibility that such DNA fragments “will integrate into the human genome” and cause cancer. As we’ve detailed , trace amounts of residual DNA are expected in vaccines, but there is no evidence the DNA can alter a person’s DNA or cause cancer.

Finally, the review highlighted findings from a Cleveland Clinic observational study that it called the “best evidence for the failure of the COVID-19 mRNA vaccine’s ability to confer protection against COVID-19.” The study, which identified a correlation between more COVID-19 vaccine doses and a higher rate of testing positive for a coronavirus infection, has frequently been cited by those opposed to vaccination. But as  we’ve   explained , the finding runs counter to that of many other studies, which have generally found increased protection with more doses. And the paper did not demonstrate that more doses actually cause an increased risk of infection. In fact, many experts suspect that the association is likely the result of other differences between people who received a different number of doses. Moreover, the primary purpose of vaccination is to protect against severe disease — and there is abundant evidence that the COVID-19 vaccines have been very successful on that front.

“Lessons learned? More like conspiracies spun,” wrote surgical oncologist Dr.  David Gorski  in a  post  about the review in his blog Respectful Insolence.

The authors of the review have also been criticized for citing their own studies in the review and for including non-scientific publications as primary sources. 

“BTW, the McCullough, Kirsch, etc. Cureus paper that is purportedly a scientific review article references trialsitetnews, epoch times, brownstone, the spectator, children’s health defense, and conservative review as primary sources for some of their points, as well as 11 substack articles/blogs, a youtube/twitter video, and 2 explicit anti-vaccine books, plus a large number of self-citations from the review authors,”  Jeffrey S. Morris , director of the division of biostatistics in the department of biostatistics, epidemiology and informatics at the University of Pennsylvania Perelman School of Medicine,  wrote on X  on Feb. 1.

Peer Review Doesn’t Guarantee Scientific Quality

Much of the complimentary coverage of the review paper by some of the usual misinformation spreaders has emphasized that it was published in a peer-reviewed journal.

“A review paper published last week in the journal Cureus is the first peer-reviewed paper to call for a global moratorium on the COVID-19 mRNA vaccines,” declared a Jan. 29  article  published on Robert F. Kennedy’s anti-vaccine website, Children’s Health Defense. The story also received attention on  social media .

Peer review , or the process of having fellow scientists provide feedback on a manuscript and whether it is good enough to publish, can be immensely helpful in ensuring that a given paper does not contain major flaws or errors. But peer review is only as good as the feedback provided — and it does not automatically mean the paper can be trusted. Nor are all peer-reviewed journals  the same , since each has different standards and reputations.

Cureus is unusual in that it focuses on publishing papers quickly and advertises “efficient” peer review and a “hassle-free” publishing experience. The journal’s metrics for the last six months indicate that the average time from submission to publication is 33 days and that the acceptance rate is 51%. For context, the prestigious journal  Nature — which some posts have misleadingly likened Cureus to, as they share the same parent publisher — has a median time of 267 days for submission to acceptance and an 8%  acceptance rate . Per the  article information  for this review paper, the peer-review process took  77  days.

In 2015, responding to concerns about the journal and its fast peer-review process, the founder, president and co-editor-in-chief of Cureus, Dr. John R. Adler, said that “by design peer rejection is not a big part of our review process,” and that the journal also relies on post-publication review to “sort out what is quality/important.”

A paper by Emory University librarians that was presented at a 2022 conference classified Cureus as potentially “untrustworthy or predatory.” The journal is available on PubMed Central, the National Institutes of Health’s database of biomedical research, but is not indexed on MEDLINE, which requires some vetting for inclusion. (A paper’s appearance in either database does not imply any kind of endorsement by the NIH.)

Cureus, notably, published  two problematic  studies about ivermectin for COVID-19 in 2022. As we reported at the time, the results of the studies were inconsistent with stronger studies that did not find any benefit of using ivermectin for COVID-19. Both studies had methodological flaws and were authored by ivermectin activists —  a fact that was not disclosed  at the time of publication.

Although even the best journals occasionally retract published studies, Cureus has ended up  multiple   times  in the pages of Retraction Watch, a blog and online database of retractions — most recently on Jan. 26 for  56 studies  retracted for faked authorship nearly two years after they were first flagged. In 2022, Retraction Watch  reported  that a study retracted by Frontiers in Medicine was later updated and published in Cureus.

Editor’s note: SciCheck’s articles providing accurate health information and correcting health misinformation are made possible by a grant from the Robert Wood Johnson Foundation. The foundation has no control over FactCheck.org’s editorial decisions, and the views expressed in our articles do not necessarily reflect the views of the foundation.

“ Safety of COVID-19 Vaccines .” CDC. Accessed 15 Feb 2024. 

“ Selected Adverse Events Reported after COVID-19 Vaccination ”. CDC. Accessed 15 Feb 2024. 

“ COVID-19 Vaccine Effectiveness Update .” CDC. Accessed 15 Feb 2024.

Van Beusekom, Mary. “ Global COVID vaccination saved 2.4 million lives in first 8 months, study estimates .” CIDRAP, University of Minnesota. 31 Oct 2023. 

Watson, Oliver J., et al. “ Global impact of the first year of COVID-19 vaccination: a mathematical modelling study .” Infectious Diseases. 23 Jun 2022. 

Trang, Brittany. “ Covid vaccines averted 3 million deaths in U.S., according to new study .” Stat. 13 Dec 2022. 

“ COVID-19 vaccinations have saved more than 1.4 million lives in the WHO European Region, a new study finds .” WHO. Press release. 16 Jan 2024. 

Mead, M. Nathaniel, et al. “ COVID-19 mRNA Vaccines: Lessons Learned from the Registrational Trials and Global Vaccination Campaign .” Cureus. 24 Jan 2024. 

Yandell, Kate. “ Tucker Carlson Video Spreads Falsehoods on COVID-19 Vaccines, WHO Accord .” FactCheck.org. 12 Jan 2024. 

Yandell, Kate. “ Faulty Science Underpins Florida Surgeon General’s Call to Halt mRNA COVID-19 Vaccination .” FactCheck.org. 5 Jan 2024. 

Jaramillo, Catalina. “ mRNA Vaccines Protect Against COVID-19 Mortality, Contrary to Misleading Posts .” FactCheck.org. 26 May 2023. 

Jaramillo, Catalina. “ COVID-19 Vaccine Benefits Outweigh Small Risks, Contrary to Flawed Claim From U.K. Cardiologist .” FactCheck.org. 8 May 2023. 

Yandell, Kate. “ Cleveland Clinic Study Did Not Show Vaccines Increase COVID-19 Risk .” FactCheck.org. 16 Jun 2023. 

Jaramillo, Catalina. “ Autopsy Study Doesn’t Show COVID-19 Vaccines Are Unsafe .” FactCheck.org. 21 Dec 2022. 

Swann, Sara. “ Experts say mRNA COVID-19 vaccines have saved millions of lives, not caused mass deaths .” PolitiFact. 9 Feb 2024. 

Wong, Adrian. “ COVID-19 mRNA Vaccines Lessons Learned Fact Check! ” Techarp. 30 Jan 2024. 

Gorski, David. “ Antivaxxers write about “lessons learned” but know nothing .” Respectful Insolence. 26 Jan 2024. 

McDonald, Jessica. “ Flawed Analysis of New Zealand Data Doesn’t Show COVID-19 Vaccines Killed Millions .” FactCheck.org. 15 Dec 2024. 

Yandell, Kate. “ COVID-19 Vaccines Save Lives, Are Not More Lethal Than COVID-19 .”  FactCheck.org. 6 Nov 2023. 

Yandell, Kate. “ Posts Push Unproven ‘Spike Protein Detoxification’ Regimen .” FactCheck.org. 21 Sep 2023. 

Jaramillo, Catalina. “ Clinical Trials Show Ivermectin Does Not Benefit COVID-19 Patients, Contrary to Social Media Claims .” FactCheck.org. 15 Sep 2022. 

Robertson, Lori. “ No New Revelation on Hydroxychloroquine and COVID-19 .” FactCheck.org. 2 Jul 2021. 

McDonald, Jessica. “ COVID-19 Vaccination Increases Immunity, Contrary to Immune Suppression Claims .” FactCheck.org. 29 Jul 2022. 

Gorski, David. “ 2021: The year the weaponization of VAERS went mainstream .” Respectful Insolence. 27 Dec 2021. 

McDonald, Jessica. “What VAERS Can and Can’t Do, and How Anti-Vaccination Groups Habitually Misuse Its Data.” FactCheck.org. 6 Jun 2023. 

Fraiman, Joseph. “ Serious adverse events of special interest following mRNA COVID-19 vaccination in randomized trials in adults .” Vaccine. 22 Sep 2022. 

McDonald, Jessica, and Catalina Jaramillo. “ DeSantis’ Dubious COVID-19 Vaccine Claims .” FactCheck.org. 2 May 2023. 

Black, Steven, and Stephen Evans. “ Serious adverse events following mRNA vaccination in randomized trials in adults .” Vaccine. 26 May 2023. 

Yandell, Kate. “ Posts Spread False Claim About Moderna Patent Application .” FactCheck.org. 22 Nov 2023. 

Yandell, Kate. “ COVID-19 Vaccines Have Not Been Shown to Alter DNA, Cause Cancer .” FactCheck.org. 26 Oct 2023. 

Jaramillo, Catalina, and Kate Yandell. “ RFK Jr.’s COVID-19 Deceptions .” FactCheck.org. 11 Aug 2023. 

Morris, Jeffrey (@jsm2334) “ BTW, the McCullough, Kirsch, etc. Cureus paper that is purportedly a scientific review article references trialsitetnews, epoch times, brownstone, the spectator, children’s health defense, and conservative review as primary sources for some of their points, as well as 11 substack articles/blogs, a youtube/twitter video, and 2 explicit anti-vaccine books, plus a large number of self-citations from the review authors. ” X. 1 Feb 2024. 

“ Scrutinizing science: Peer review. ” Understanding Science 101. Accessed 15 Feb 2024. 

Crossley, Merlin. “ When to trust (and not to trust) peer reviewed science .” The Conversation. 12 Jul 2018.

“ Reviewer Guide .” Cureus. Accessed 15 Feb 2024. 

“ About Cureus .” Cureus. Accessed 15 Feb 2024. 

“ Journal Metrics .” Nature. Accessed 15 Feb 2024. 

“ Editorial criteria and processes .” Nature. Accessed 15 Feb 2024. 

Sparks, Katie, and Kimberly R. Powell. “ Assessing Predatory Journal Publishing Within Health Sciences Authors .” SLA conference. 31 Jul 2022.  

“ MEDLINE, PubMed, and PMC (PubMed Central): How are they different? ” NIH. 28 Dec 2023.

“ Disclaimer .” National Library of Medicine. Accessed 15 Feb 2024. 

“ Some Strange Goings On at Cureus .” Emerald City Journal. 20 Aug 2016.

Oransky, Ivan. “ Journal retracts more than 50 studies from Saudi Arabia for faked authorship .” Retraction Watch. 26 Jan 2024. 

Kincaid, Ellie. “ Researcher attacks journal for retracting his paper on COVID-19 drug .” Retraction Watch. 26 Jan 2024. 10 Jun 2022.

Jaramillo, Catalina. “ Evidence Still Lacking to Support Ivermectin as Treatment for COVID-19 .” FactCheck.org. 6 Jun 2022.

Jaramillo, Catalina. “ Clinical Trials Show Ivermectin Does Not Benefit COVID-19 Patients, Contrary to Social Media Claims .” FactCheck.org. 15 Sep 2022.

Kerr, Lucy, et al. “ Correction: Ivermectin Prophylaxis Used for COVID-19: A Citywide, Prospective, Observational Study of 223,128 Subjects Using Propensity Score Matching .” Cureus. 24 Mar 2022. 

The post Review Article By Misinformation Spreaders Misleads About mRNA COVID-19 Vaccines  appeared first on FactCheck.org .

Review Article By Misinformation Spreaders Misleads About mRNA COVID-19 Vaccines 

share this!

February 19, 2024 report

This article has been reviewed according to Science X's editorial process and policies . Editors have highlighted the following attributes while ensuring the content's credibility:

fact-checked

trusted source

AI-generated disproportioned rat genitalia makes its way into peer-reviewed journal

by Bob Yirka , Phys.org

rat anatomy

The editors at the journal Frontiers in Cell and Developmental Biology have retracted a paper after it was pointed out to them by readers that supporting images had been generated improperly by an AI image generator. In their retraction, the editors report that the reason for the retraction was that "concerns were raised regarding the nature of its AI-generated figures."

In the article, which involved research surrounding stem cells in small mammals , the authors included images depicting rat anatomy that an AI system had clearly created. In one picture, a single rat appeared to have a penis and testicles that were larger than the rest of its body—not something that occurs in nature. Some of the accompanying text was also incomprehensible. Another image showed a rat cell that did not resemble the true structure of a rat cell.

The disproportioned images in the paper are likely to add to ongoing discussions in the science community surrounding the use of AI in generating text or imagery for use in technical papers—particularly those published in established journals.

In this case, it is not clear how such problematic images wound up in a peer-reviewed journal. The authors, a combined team from Hong Hui Hospital and Jiaotong University in China, did not try to hide the fact that they had used AI to create the images ; they even credited Midjourney.

Some in the press have noted that Frontiers has a policy that allows for the use of AI-generated materials as long as their use is disclosed, which was the case in this instance. But the policy also notes that attempts must be made to fact-check anything produced by such systems, which clearly was not the case in this mix-up.

The editors at Frontiers initially posted a note on the paper claiming that the article had been corrected and that a new version would be published in short order. Not much later, the paper was retracted.

The mistakes made by the authors of the paper and the team at the journal that approved its publication are likely to be the first of many to come, though it is still not clear what changes will be required to prevent such mistakes from happening in the future.

© 2024 Science X Network

Explore further

Feedback to editors

peer review 03

Using CRISPR gene editing to prevent pigs from getting blue-ear pig disease

7 minutes ago

peer review 03

How did a tiny bee get to French Polynesia? Eight new species help solve a scientific mystery

8 hours ago

peer review 03

Saturday Citations: The neurology of pair bonding and one small step for robots

Feb 24, 2024

peer review 03

Global warming found to increase the diversity of active soil bacteria

Feb 23, 2024

peer review 03

Study shows cloud clustering causes more extreme rain

peer review 03

Scientists closer to finding quantum gravity theory after measuring gravity on microscopic level

peer review 03

Mindfulness at work protects against stress and burnout, study finds

peer review 03

Researchers develop a computer from an array of VCSELs with optical feedback

peer review 03

How to build your own robot friend: Making AI education more accessible

peer review 03

Forever chemicals reach extraordinary levels in wildlife at Holloman Air Force Base

Relevant physicsforums posts, energy consumed by weight of gear on a multiday hike.

19 hours ago

Color Recognition: What we see vs animals with a larger color range

Feb 21, 2024

Mathematical process for protein folding

Long-term effects/risks/vulnerabilities of having had covid, left atrial appendage (laa) closure for prophylactic a-fib treatment.

Feb 17, 2024

PFAS and Power Lines Cause Cancer?

Feb 16, 2024

More from Biology and Medical

Related Stories

peer review 03

Retracted anti-abortion paper contained undisclosed conflicts of interest

Jan 11, 2023

peer review 03

Researcher develops filter to tackle 'unsafe' AI-generated images

Nov 13, 2023

peer review 03

OpenAI announces Point-E, a machine learning system that quickly creates 3D images from a text prompt

Dec 21, 2022

peer review 03

Top science publisher withdraws flawed climate study

Aug 24, 2023

peer review 03

Top science editor defends peer-review system in climate row

Sep 15, 2023

peer review 03

Paper by team claiming to have achieved superconductivity at room temperature retracted

Sep 29, 2022

Recommended for you

peer review 03

40 years of crop research shows inequities

Feb 22, 2024

peer review 03

Unpacking social equity from biodiversity data: An interdisciplinary policy perspective

Jan 16, 2024

peer review 03

A whiff of tears reduces male aggression, says study

Dec 25, 2023

peer review 03

Solicitor in 19th-century Tasmania traded human Aboriginal remains for scientific accolades, study reveals

Nov 28, 2023

peer review 03

How larger body sizes helped the colonizers of New Zealand

Jul 13, 2023

peer review 03

Study examines centuries of identity lost because of slavery

Jul 6, 2023

Let us know if there is a problem with our content

Use this form if you have come across a typo, inaccuracy or would like to send an edit request for the content on this page. For general inquiries, please use our contact form . For general feedback, use the public comments section below (please adhere to guidelines ).

Please select the most appropriate category to facilitate processing of your request

Thank you for taking time to provide your feedback to the editors.

Your feedback is important to us. However, we do not guarantee individual replies due to the high volume of messages.

E-mail the story

Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient's address will be used for any other purpose. The information you enter will appear in your e-mail message and is not retained by Phys.org in any form.

Newsletter sign up

Get weekly and/or daily updates delivered to your inbox. You can unsubscribe at any time and we'll never share your details to third parties.

More information Privacy policy

Donate and enjoy an ad-free experience

We keep our content available to everyone. Consider supporting Science X's mission by getting a premium account.

E-mail newsletter

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • v.2(6); 2022 Nov

Effective Peer Review: Who, Where, or What?

Peer review is widely viewed as one of the most critical elements in assuring the integrity of scientific literature ( Baldwin, 2018 ; Smith, 2006 ). Despite the widespread acceptance and utilization of peer review, many difficulties with the process have been identified ( Hames, 2014 ; Horrobin, 2001 ; Smith, 2006 ). One of the primary goals of the peer review process is to identify flaws in the work and, by so doing, help editors choose which manuscripts to publish. It is surprising that one of the persistent problems in peer review is assessing the quality of the reviews. Both authors and journal editors expect peer review to detect errors in experimental design and methodology and to ensure that the interpretation of the findings is presented in an objective and thoughtful manner. In traditional peer review, two or more reviewers are asked to evaluate a manuscript on the basis of the expectation that if the two reviewers agree on the quality of the submission, the likelihood of a high-quality review is increased. Unfortunately, studies have not consistently confirmed a high degree of agreement among reviewers. Rothwell and Martynn (2000) evaluated the reproducibility of peer review in neuroscience journals and meeting abstracts and found that agreement was approximately what would be expected by chance. Similarly, Scharschmidt et al. (1994) found similar results in the evaluation of 1,000 manuscripts submitted to the Journal of Clinical Investigation, where clustering of grades in the middle resulted in an agreement being “…only marginally…” better than chance. These observations suggest that we cannot rely on the agreement of reviewers to be an indication of the quality of the reviews. Another potential way to evaluate the quality of reviews would be to assess the ability of reviewers to detect errors in submissions. It is generally accepted that detection of intentional fraud is beyond the scope of typical peer review, but we do expect reviewers to detect major and minor errors as a primary function of the traditional peer review system ( Hwang, 2006 ; Weissman, 2006 ). Schroter et al. (2008) evaluated the ability of reviewers to detect major and minor errors by introducing errors into three previously published papers describing randomized controlled clinical trials. Reviewers detected approximately three of the nine errors introduced in each manuscript. Unfortunately, reviewers who had undergone training in how to conduct a high-quality peer review were not significantly better than untrained reviewers. Similar results have been reported by Godlee et al. (1998) and Baxt et al. (1998) . Baxt et al. (1998) did report that reviewers who rejected or suggested revision of a manuscript identified more errors than those who accepted the manuscript (decision: 17.3% of major errors detected [accept], 29.6% of major errors detected [revise], and 39.1% of major errors detected [reject]). It is almost certainly true that the extent of the failure to recognize errors in submitted manuscripts may differ among scientific disciplines and journals. It also however seems likely that these observations do have some applicability to journals such as JID Innovations . It is critical that both authors and editors are cognizant of these limitations of peer review in their assessment of reviews. These findings compel journals to continue to work to develop new strategies to train and evaluate reviewers. The findings also suggest that factors beyond the failure to detect objective mistakes in a manuscript may be playing a role in the discrepancy in reviewers’ evaluations. One area of ongoing concern in the peer review process is the role of reviewer bias in assessing the scientific work of colleagues ( Kuehn, 2017 ; Lee et al, 2013 ; Tvina et al, 2019 ).

Bias in the peer review process can take many forms, including collaborator/competitor bias, affiliation bias based on an investigator’s institution or department, geographical bias based on the region or country of origin, racial bias, and gender or sex bias ( Kuehn, 2017 ; Lee et al, 2013 ; Tvina et al, 2019 ). All of these forms of bias present the risk that a decision of the reviewer will not be based solely on the quality or merit of the work but rather be influenced by a bias of the reviewer. We and other journals routinely seek to avoid selecting individuals to review work from their own institutions and ask all reviewers to declare any potential personal conflicts of interest. All these methods require either the editor or the reviewer to identify a bias and fail to address the issue of implicit or unconscious reviewer bias. The dominant method currently utilized for peer review is the so-called single-blind review, in which the identity and affiliations of the authors are known to the reviewers, whereas the identity of reviewers remains unknown to the authors. This has led to concern that knowledge of the identity of the authors and their institutions may be the source of significant reviewer bias, especially implicit bias, in the evaluation of manuscripts. Double anonymized peer review (DAPR), also known as double-blind peer review, has been suggested as a way to address this issue ( Bazi, 2020 ; Lee et al, 2013 ). Studies have compared single-blind with double-blind reviewing and reported that there is no significant difference in the quality of the reviews ( Alam et al, 2011 ; Godlee et al, 1998 ; Justice et al, 1998 ; van Rooyen et al, 1998 ). Although these studies looked at measures such as the number of errors detected, acceptance rate, and distribution of initial reviewer scores, they were not designed to address specific sources of bias such as authors’ gender, institution, or geographic location. Other studies have been undertaken to directly address the issue of bias in the peer review process. Ross et al (2006) compared the acceptance of abstracts submitted to the American Heart Association’s annual scientific meeting during a period when the reviewers knew the identity and origin of the authors (i.e., single-blind review) with when this information was not known by the reviewers (i.e., double-anonymized peer review). They found a significant increase in acceptance of non‒United States abstracts and abstracts from non-English speaking countries when the reviewers were unaware of the country of origin of the abstracts ( Ross et al, 2006 ). They also found a significant decrease in the acceptance of abstracts from prestigious institutions when the reviewers were unaware of the institutions where the work was done. In a similar study, Tomkins et al. (2017) found that papers submitted to a prestigious computer science meeting were more likely to be accepted if they were from famous authors, top universities, and top companies. Okike et al. (2016) documented similar results for manuscripts submitted to the orthopedic literature. They submitted a fabricated manuscript that was presented as being written by two prominent orthopedic surgeons (past Presidents of the American Academy of Orthopedic Surgeons) from prestigious institutions. When reviewed in the traditional single-blind fashion, which included the identity of the authors, the manuscript was accepted by 87% of the reviewers. By contrast, when the identity of the authors was unknown, the manuscript was accepted by 68% of the reviewers ( P  = 0.02) ( Blank, 1991 ). A study conducted at The American Economic Review found that authors at near-top-ranked universities experienced lower acceptance rates when authorship was anonymized ( Blank, 1991 ). Of interest, they also found that for women, there was no difference in the acceptance rate between the double-anonymized and single-blinded reviews; however, for men, the acceptance rate was lower with double-anonymized reviews.

These studies provide strong evidence that knowledge of who and where the study was performed can impact the acceptance of abstracts and manuscripts. This conflicts with the goal of the review process to base our judgments on the quality of what the results demonstrate. It is difficult to estimate how much this may affect the fate of a manuscript at JID Innovations . We do not have evidence that our review process has been impacted by bias as is reported in the studies discussed. However, neither can we state with certainty that such bias is not a factor in the reviews we receive. One of the goals of JID Innovations is to be a truly open-access journal available to all investigators in skin science from around the world. We have sought to be an outlet for studies that challenge existing paradigms or that may report negative results. We want to be seen as providing fair and objective reviews for all authors, regardless of where they work or who they are. If we are to achieve this goal, it is imperative that the who and where of a specific manuscript do not negatively impact the evaluation of the what. We want young investigators, investigators at less prestigious institutions or from less well-known laboratories, and investigators from any country around the world to be confident that their work will be judged by what they report and not by the who and the where.

To be true to this mission, JID Innovations will be initiating DAPR starting in October 2022. This is not being done because we are aware of any issues of bias with our current process of peer review but because we realize that the absence of proof is not proof of absence. As a part of this process, authors will be asked to remove identifying material from manuscripts at the time of submission in preparation for the review process ( https://www.jidinnovations.org/content/authorinfo ). As a result, primary reviewers will see only the what of the manuscript. We realize that this process involves extra work for both the authors and our staff, but we feel the benefits will outweigh this small cost. Indeed, in other journals that have taken this step, surveys have shown that both authors and reviewers ultimately prefer double-anonymized reviews ( Bennett et al, 2018 ; Moylan et al, 2014 ). We realize that achieving 100% anonymization of a manuscript is nearly impossible. Studies have shown that the rate of successful anonymizing, where the reviewers cannot discern the authorship of a manuscript, ranged from 47 to 73%. It is however interesting that even with this rate of success in the anonymizing process, a meta-analysis of trials of double- versus that of single-blind peer review has suggested an impact, with lower acceptance rates with double-anonymized peer review ( Ucci et al, 2022 ). More work clearly needs to be done to assess the value of the DAPR process, and we will be monitoring our results carefully.

The institution of DAPR in JID Innovations will assure our authors that the what of their manuscript is our focus. It does not matter who you are or where you are from. It will also emphasize to our reviewers that our focus is on the what. We will be carefully monitoring the results of this new policy and plan to report back on our experience. We also welcome your feedback on your experience as a reviewer and author for JID Innovations ; send your comments to us at [email protected] .

Finally, this decision should be seen not as the end of our efforts to improve the peer review process but merely as a first step. We will continue to work to improve all aspects of the peer review process for JID Innovations . We firmly believe that the use of double-blind -anonymized peer review will bring us closer to ensuring to our authors and readers that the work that is published by JID Innovations has been selected on the basis of what the paper reports and not on who performed the studies or where they were located.

Conflict of Interest

The author states no conflicts of interest.

  • Alam M., Kim N.A., Havey J., Rademaker A., Ratner D., Tregre B., et al. Blinded vs. unblinded peer review of manuscripts submitted to a dermatology journal: a randomized multi-rater study. Br J Dermatol. 2011; 165 563‒7. [ PubMed ] [ Google Scholar ]
  • Baldwin M. Scientific autonomy, public accountability, and the rise of “peer review” in the Cold War United States. Isis. 2018; 109 :538–558. [ Google Scholar ]
  • Baxt W.G., Waeckerle J.F., Berlin J.A., Callaham M.L. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998; 32 310‒7. [ PubMed ] [ Google Scholar ]
  • Bazi T. Peer review: single-blind, double-blind, or all the way-blind? Int Urogynecol J. 2020; 31 :481–483. [ PubMed ] [ Google Scholar ]
  • Bennett K.E., Jagsi R., Zietman A. Radiation oncology authors and reviewers prefer double-blind peer review. Proc Natl Acad Sci USA. 2018; 115 :E1940. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Blank R.M. The effects of double-blind versus single-blind reviewing: experimental evidence from the American Economic Review. Am Econ Rev. 1991; 81 :1041–1067. [ Google Scholar ]
  • Godlee F., Gale C.R., Martyn C.N. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998 280237‒40. [ PubMed ] [ Google Scholar ]
  • Hames I. Peer review at the beginning of the 21st century. Sci Ed. 2014; 1 :4–8. [ Google Scholar ]
  • Horrobin D.F. Something rotten at the core of science? Trends Pharmacol Sci. 2001; 22 51‒2. [ PubMed ] [ Google Scholar ]
  • Hwang W.S. Can peer review police fraud? Nat Neurosci. 2006; 9 :149. [ PubMed ] [ Google Scholar ]
  • Justice A.C., Cho M.K., Winker M.A., Berlin J.A., Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators [published correction appears in JAMA 1998;280:968. JAMA. 1998; 280 240‒2. [ PubMed ] [ Google Scholar ]
  • Kuehn B.M. Rooting out bias. ELife. 2017; 6 [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Lee C.J., Sugimoto C.R., Zhang G., Cronin B. Bias in peer review. JASIST. 2013; 64 :2–17. [ Google Scholar ]
  • Moylan E.C., Harold S., O'Neill C., Kowalczuk M.K. Open, single-blind, double-blind: which peer review process do you prefer? BMC Pharmacol Toxicol. 2014; 15 :55. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Okike K., Hug K.T., Kocher M.S., Leopold S.S. Single-blind vs double-blind peer review in the setting of author prestige. JAMA. 2016; 316 1315‒6. [ PubMed ] [ Google Scholar ]
  • Ross J.S., Gross C.P., Desai M.M., Hong Y., Grant A.O., Daniels S.R., et al. Effect of blinded peer review on abstract acceptance. JAMA. 2006; 295 1675‒80. [ PubMed ] [ Google Scholar ]
  • Rothwell P.M., Martyn C.N. Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain. 2000; 123 1964‒9. [ PubMed ] [ Google Scholar ]
  • Scharschmidt B.F., DeAmicis A., Bacchetti P., Held M.J. Chance, concurrence, and clustering. Analysis of reviewers' recommendations on 1,000 submissions to the Journal of Clinical Investigation. J Clin Invest. 1994; 93 1877‒80. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Schroter S., Black N., Evans S., Godlee F., Osorio L., Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med. 2008; 101 507‒14. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006; 99 178‒82. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Tomkins A., Zhang M., Heavlin W.D. Reviewer bias in single- versus double-blind peer review. Proc Natl Acad Sci USA. 2017; 114 12708‒13. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Tvina A., Spellecy R., Palatnik A. Bias in the peer review process: can we do better? Obstet Gynecol. 2019; 133 1081‒3. [ PubMed ] [ Google Scholar ]
  • Ucci M.A., D'Antonio F., Berghella V. Double- vs single-blind peer review effect on acceptance rates: a systematic review and meta-analysis of randomized trials. Am J Obstet Gynecol MFM. 2022; 4 [ PubMed ] [ Google Scholar ]
  • van Rooyen S., Godlee F., Evans S., Smith R., Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998; 280 234‒7. [ PubMed ] [ Google Scholar ]
  • Weissmann G. Science fraud: from patchwork mouse to patchwork data. FASEB J. 2006; 20 587‒90. [ PubMed ] [ Google Scholar ]

Frontiers in Cell and Developmental Biology / Futurism

Scientific Article With Insane AI-Generated Images Somehow Passes Peer Review

W-what is that.

The questionable use of AI-generated content in academic journals poses a problem for the credibility of the scientific community. Some of the culprits are easier to weed out than others, but no one could expect an example as ridiculous as this latest case.

As spotted by a user on X-formerly-Twitter , a new study published this week in the journal Frontiers in Cell and Developmental Biology   — which is supposedly peer-reviewed , mind you — heavily features figures and diagrams that are obviously AI-generated. And, how should we put this? They're insane-looking nonsense.

Take a look, for example, at this diagram of an extremely well endowed mouse. The morphology is so grotesque and comical it defies any explanation, flaunting what appears to be a dissected view of testicles as nearly as large as the mouse itself. They're also protruding from its stomach and just sort of hanging out there in the open, tethered to some unknown upper anatomy out of the frame.

Note the emphasis on "appears to be," because the labels on the diagram are made up gobbledygook like "testtomcels," "retat," and "dck." But it did get a few things right though, like this helpful line pointing to the middle of the creature labeled "Rat."

Article published a couple of days ago. Every figure in the article is AI generated and totally incomprehensible. This passed "peer-review" https://t.co/8oeyjbn098 pic.twitter.com/d58ME9bXTk — clifford (@cliff_swan) February 15, 2024

To the researchers' credit, they weren't being deceptive, disclosing that the "images in this article were generated by Midjourney." Yet they appear to have made little or no effort to edit them, which ends up undermining their credibility anyway. Moral of the story: double check your AI-hallucinated stuff.

Speaking of double-checking, some members in the scientific community warn that Frontiers is what's known as a " predatory journal ," a type of publication that misleadingly claims to peer-review submissions while taking advantage of scientists desperate to get a paper published. If true, that might help explain why these absolutely confounding images didn't raise any alarms; no one, it seems, is doing any real checking.

It's worth noting, however, that not everyone agrees with that characterization of Frontiers,  as what journals are considered "predatory" remains a heated topic in the community. Addressing this most recent controversy, the publisher acknowledged that it's seen the "scrutiny" blowing up on social media.

"We thank the readers for their scrutiny of our articles: when we get it wrong, the crowdsourcing dynamic of open science means that community feedback helps us to quickly correct the record,"  Frontiers wrote on X .

Regardless of the publication, it's undeniable that generative AI has made worrying inroads into academia. Some papers have been more upfront by listing ChatGPT as a co-author . Others made no such disclosure, but managed to slip through peer review anyway .

The ones that do get caught often make careless mistakes , like forgetting to delete telltale AI phrases such as "regenerate response." In any case, the content an AI generates is usually filled with factual errors . That's a no-go for any kind of published material, but especially the scientific kind, which relies on highly technical knowledge that a general-purpose chatbot won't use correctly.

More on AI: OpenAI Hiring Detective to Find Who's Leaking Its Precious Info

Share This Article

DISCLAIMER(S)

Articles may contain affiliate links which enable us to share in the revenue of any purchases made.

Registration on or use of this site constitutes acceptance of our Terms of Service.

© Recurrent Ventures Inc, All Rights Reserved.

IMAGES

  1. The Peer Review Process

    peer review 03

  2. The peer review process

    peer review 03

  3. 10 Ultimate Steps: How to Peer Review a Review Article

    peer review 03

  4. What is Peer Review?

    peer review 03

  5. Peer Review

    peer review 03

  6. The Peer Review Process

    peer review 03

VIDEO

  1. Author Interview with Vincent Krivda

  2. Peer review practice round

  3. Peer Review Feedback

  4. Peer Review

  5. Peer To Peer Podcast Is ENDING... RIP

  6. Webinar : Introduction to Peer Review System

COMMENTS

  1. M-05-03, Peer Review

    December 16, 2004 MEMORANDUM FOR HEADS OF DEPARTMENTS AND AGENCIES OMB has today issued a bulletin applicable to all departments and agencies entitled "Final Information Quality Bulletin for Peer Review." This Bulletin establishes government-wide guidance aimed at enhancing the practice of peer review of government science documents.

  2. Peer review guidance: a primer for researchers

    The International Committee on Medical Journal Editors (ICMJE) qualifies peer review as a continuation of the scientific process that should involve experts who are able to timely respond to reviewer invitations, submitting unbiased and constructive comments, and keeping confidentiality [ 8 ].

  3. Peer Review in Scientific Publications: Benefits, Critiques, & A

    Peer Review is defined as "a process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field" ( 1 ). Peer review is intended to serve two primary purposes.

  4. A step-by-step guide to peer review: a template for patients and novice

    The peer review template for patients and novice reviewers ( table 1) is a series of steps designed to create a workflow for the main components of peer review. A structured workflow can help a reviewer organise their thoughts and create space to engage in critical thinking. The template is a starting point for anyone new to peer review, and it ...

  5. How to write a peer review

    Co-reviewing (sharing peer review assignments with senior researchers) is one of the best ways to learn peer review. It gives researchers a hands-on, practical understanding of the process. In an article in The Scientist , the team at Future of Research argues that co-reviewing can be a valuable learning experience for peer review, as long as ...

  6. How to Write a Peer Review

    Here's how your outline might look: 1. Summary of the research and your overall impression. In your own words, summarize what the manuscript claims to report. This shows the editor how you interpreted the manuscript and will highlight any major differences in perspective between you and the other reviewers. Give an overview of the manuscript ...

  7. How to Write a Peer Review: 12 things you need to know

    3) Skim the paper very quickly to get a general sense of the article. Underline key words and arguments, and summarise key points. This will help you quickly "tune in" to the paper during the next read. 4) Sit in a quiet place and read the manuscript critically. Make sure you have the tables, figures and references visible.

  8. Open Peer Review

    Peer review is a pillar of scientific communication, the mechanism we rely on to ensure that published research is thoroughly vetted and scientifically valid. For that reason, we tend to think of peer review as a monolith-iconic, stable, and consistent. In fact, journals use many different forms and applications of peer review, often in parallel.

  9. Reviewers

    Tools and resources Background Reviewers play a pivotal role in scholarly publishing. The peer review system exists to validate academic work, helps to improve the quality of published research, and increases networking possibilities within research communities.

  10. Research Methods: How to Perform an Effective Peer Review

    Peer review has been a part of scientific publications since 1665, when the Philosophical Transactions of the Royal Society became the first publication to formalize a system of expert review. 1,2 It became an institutionalized part of science in the latter half of the 20 th century and is now the standard in scientific research publications. 3 In 2012, there were more than 28 000 scholarly ...

  11. Peer review

    Abstract Peer review has a key role in ensuring that information published in scientific journals is as truthful, valid and accurate as possible. It relies on the willingness of researchers to give of their valuable time to assess submitted papers, not just to validate the work but also to help authors improve its presentation before publication.

  12. How to Peer Review

    When peer reviewing, it is helpful to think from the point of view of three different groups of people: Authors. Try to review the manuscript as you would like others to review your work. When you point out problems in a manuscript, do so in a way that will help the authors to improve the manuscript. Even if you recommend to the editor that the ...

  13. Understanding the peer review process: A step-by-step guide for

    Step 1: Submission The first step in the peer review process begins with the submission of a research manuscript to a scholarly journal. Researchers should carefully select a journal that aligns with the scope and focus of their study. It is essential to review the journal's guidelines for authors and formatting requirements to ensure compliance.

  14. Everything You Need to Know About Peer Review

    Guides to Good Reviewing are widely available, both generally (Box 1) and specifically, for individual journals, and for specific types of submissions.For example, Moher offered optimal strategies to consider when peer reviewing a systematic review and meta-analysis, recommending the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement [11], [12].

  15. What Is Peer Review?

    Peer review, sometimes referred to as refereeing, is the process of evaluating submissions to an academic journal. Using strict criteria, a panel of reviewers in the same subject area decides whether to accept each submission for publication.

  16. PDF Clarified AICPA Standards for Performing and Reporting on Peer Reviews

    Concepts Common to All Peer Reviews. 8 . Acceptance date. A peer review is accepted on the date the peer review committee (the committee) or report acceptance body (RAB) concludes a peer review was performed and reported in on accordance with the standards or, for . pass with deficiencies. or . fail. reports, when the reviewed

  17. 5.3: Peer Review as Collaboration

    5.3: Peer Review as Collaboration. One of the most common types of collaboration done in writing classes comes in the form of in-class "group work" or as peer review sessions. Peer review has become a common practice for contemporary composition and rhetoric classrooms. Basically, it is the process where small groups of students read ...

  18. The peer review process

    Today, peer review continues to evolve with the introduction of open review (reviewer comments posted publicly with the final article), postpublication review (reviews solicited from readers in an open forum after article publication), and journal review networks (where reviews are transferred from one journal to another when an article is rejec...

  19. PDF Peer Review: benefits, perceptions and alternatives

    Peer review is widely supported by academics, who overwhelming (93%) disagreed in our survey that peer review is unnecessary. The large majority (85%) agreed that peer review greatly helps scientific communication and believed (83%) that without peer review there would be no control. 3. Peer review improves the quality of the published paper ...

  20. Peer review isn't perfect − I know because I teach others how to do it

    As a process, peer review evaluates academic papers for their quality, integrity and impact on a field, largely shaping what scientists accept as "knowledge." By instinct, any academic follows...

  21. Perspectives on peer review: insights from industry experts

    By Jin Zhang Celebrating Peer Review Week 2023 with Elsevier The landscape of academic publishing, along with the role of peer review, is undergoing rapid change, driven by new policies, business models, technologies, and the need for greater transparency and reproducibility.

  22. The future of peer review in the 17th Century

    The insightful collection of referee reports sheds light on the evolving peer review process at the Royal Society from 1832-1949. Open and shut. Taking inspiration from other European science academies at the time, the Royal Society's editorial committee made an early endeavour into open peer review in 1831-32. In the same period that saw the ...

  23. How to Navigate the Peer Review Process

    She manages manuscript submissions and review article commissions across all areas of neuroscience and oversees the peer-review process. Her scientific training is in mouse genetics, behavior, and molecular and developmental neurobiology. Before joining Neuron, Mariela was a postdoc at Harvard. She completed her Ph.D. in Neurobiology at Caltech ...

  24. Scientists aghast at bizarre AI rat with huge genitals in peer-reviewed

    Many researchers expressed surprise and dismay that such a blatantly bad AI-generated image could pass through the peer-review system and whatever internal processing is in place at the journal.

  25. Review Article By Misinformation Spreaders Misleads About mRNA ...

    In 2015, responding to concerns about the journal and its fast peer-review process, the founder, president and co-editor-in-chief of Cureus, Dr. John R. Adler, said that "by design peer ...

  26. AI-generated disproportioned rat genitalia makes its way into peer

    Top science editor defends peer-review system in climate row. Sep 15, 2023. Paper by team claiming to have achieved superconductivity at room temperature retracted. Sep 29, 2022.

  27. Institution of Engineering and Technology

    Are you interested in publishing your original research papers on the latest developments across all electronic and electrical engineering related fields? If yes, then you should check out the author guidelines of Electronics Letters, a Gold Open Access rapid-communication journal from the Institution of Engineering and Technology. You will find detailed information on the submission process ...

  28. Effective Peer Review: Who, Where, or What?

    In traditional peer review, two or more reviewers are asked to evaluate a manuscript on the basis of the expectation that if the two reviewers agree on the quality of the submission, the likelihood of a high-quality review is increased. Unfortunately, studies have not consistently confirmed a high degree of agreement among reviewers.

  29. Scientific Article With Insane AI-Generated Images Somehow Passes Peer

    Others made no such disclosure, but managed to slip through peer review anyway. The ones that do get caught often make careless mistakes , like forgetting to delete telltale AI phrases such as ...