How does the journal editor choose suitable peer reviewers?
When journal editors receive a paper for consideration, they typically send it to two or three academics who are experts in the field. Journals in the field of sport management usually use only two PRs, unless the initial round of reviews raises differing opinions on the work, and the field of social sciences sometimes invites four to participate in the initial round.
Choosing suitable PRs for can be a very time-consuming and laborious task for journal editors. To simplify the task, some journal editors invite authors to nominate PRs who have expertise that is specific to their manuscripts. While some authors avoid this in case their suggestion is ignored, according to John E. Dolbow (2013), Professor of Civil and Environmental Engineering at Duke University and journal editor of Finite Elements in Analysis and Design, provided you give adequate justification, the journal editor will typically include your suggestion as one of the selected PRs. In her capacity as an author, Geraldine sometimes takes the opportunity to “nominate an expert in the field, whose work I am building on.” Thus, it is worth remembering that taking the time to suggest good reviewers for your manuscripts will greatly facilitate the review process.
To speed up the process of selecting suitable PRs, many journals have adopted electronic peer-review systems; however, these have seen an increase in the amount of ethical problems caused by “fake reviewers.” In such cases, authors suggest seemingly legitimate academic reviewers, but the webmail addresses that accompany the suggestions are controlled either by a biased close associate of the author or the author of the manuscript posing as a qualified reviewer (Doffegnies & Haire, 2016).
To overcome these problems, ScholarOne Manuscript, a journal and peer review tool, has implemented a Reviewer Locator, which automatically searches for reviewers based on a manuscript’s keywords and abstract. The keywords and abstracts are “fed into the Web of Science and a complex algorithm discovers potential reviewers based on their previous publications in the same area of expertise” (Doffegnies & Haire, 2016). This method broadens the search for potential PRs for the journal and ensures that a wide range of experts in the field can evaluate the research being presented.
The journal editor usually asks the PR to recommend either (1) accepting the paper as it is, (2) telling the authors to make minor revisions, (3) telling the authors to make major revisions, or (4) rejecting the paper altogether.
Although the journal editors don’t tend to give the PRs specific criteria to work to, collectively, the PRs reported that they aim to answer the following questions when working on their reviews:
- Is the topic clear and relevant to the readership?
- Is the work theoretically and conceptually appropriate?
- Are the statistics and analytical and experimental methods used appropriate and correctly executed?
- Does the paper cover all the relevant literature?
- Is it well written?
- Does the author convey the information effectively?
Once all the PRs have submitted their reviews, the journal editor reads the reviews and the manuscript and makes a decision (accept/minor revision/major revision/reject) to give to the author(s). As the pie chart below shows, it is very rare for a paper to be accepted for publication without any further revisions being required by the author (1%), and the majority of papers require major revisions (50%) or are rejected immediately (40%).
Few papers receive recommendations for rejection from the PRs because most rejections are desk-rejections, which means that the journal editor decides that the paper is not relevant to the remit of the journal or that it is so clearly underdeveloped and executed that it cannot be sent to busy, volunteer PRs in its current state.
The review process is repeated until the PRs and journal editor are happy that the paper is ready to be published; most articles that are eventually accepted go through between one and three rounds of reviews. The entire process (shown below) from submitting your paper to publication can thus take up to two years in general.
What process does the peer reviewer use to conduct a review?
The PRs all follow slightly different methods of conducting a peer review. Judith reads and makes notes about what needs to be improved as she goes, and she might correct inaccuracies. She usually works at a PC but sometimes prints a hard copy to read on her commute. She typically completes a peer review within two to four hours.
When Sara receives a paper, she usually reads through a printed copy, making notes as she goes. Once she has read it and has a good idea of what she wants to say about it, she either completes the online form or compiles a Word document with comments, questions, and suggestions to submit online.
Geraldine follows a similar process to Sara and reads a printed copy of the paper thoroughly, which she annotates with extensive comments and notes as she is reading it. She usually leaves it for a day or two so that she can think about the paper and her comments before writing the review and submitting it electronically.
Stephen takes a seemingly more thorough approach than the other three PRs. Upon receiving the submission, he usually prints out a hard copy to read through the first time, as if he were not reviewing it, to get an overall sense of the ideas and topic and of the clarity with which the author(s) communicate. He then reviews to the references section and skims through to see whether the author(s) have included important scholars within their area of interest: “If not, I make a mental note, as this will require justification within.” He then reads the submission for a second time on his office computer and makes comments and tracked changes throughout. After leaving the submission for a day or two, he reads it a third time, checking over his comments as he goes, and then he begins writing his response based on his comments, including his recommendations. Finally, he reviews his response to the author(s) to ensure his tone is respectful and that his overall feedback is useful and potentially productive. Stephen thus usually spends two full days cumulatively (over three or four days) on each review.
I asked the PRs about whether they do any additional reading to understand the content of the paper. All four PRs replied that it is rare for them to do extra reading while performing a peer review because if they are reviewing a paper, it is usually because they have some expertise in the topic area. However, Stephen stated, “If the author(s) cite a study with which I am not familiar, I will find the article to ensure it supports the point being made.” Additionally, if the author(s) relies heavily on one or a few article(s), Stephen will read it (them) thoroughly. Geraldine agreed that it might be necessary to read one or two extra papers to assist in writing the review.
Despite there being several PRs for each paper, the reviews are conducted blindly, meaning the PRs are not told the identities of the authors or the other PRs, and they are thus unable to collaborate with the other PRs when conducting the review. However, it is possible that PRs could be aware of who the authors are if the community of authors is small, if they have heard about the research through the grapevine, or if they know the author’s style of writing. While there is a concern that this would affect bias, Dolbow (2013) affirmed that acquaintances can be just as brutal with reviews as anyone else.
Once the PRs have completed their review, they are asked to provide a report to the journal editor, which typically takes the form of a detailed, structured report under a number of specific headings, with the main narrative review (sometimes several pages long) being the primary focus (Judith). Additional comments that are not visible to the author(s) are usually forwarded to the journal editor, and sometimes PRs are required to rate the submission on various criteria on a given scoresheet.
The PRs are generally required to complete a web-based form on an academic journal interface designed specifically for the process of submitting and reviewing papers, such as ScholarOne Manuscript or Manuscript Central. These interfaces allow accompanying files to be uploaded, e.g., a Word document that includes detailed comments. However, some journals still conduct the process via email, which is clearly less efficient, and different systems require different things, i.e., some ask for an evaluation on a number of key criteria using a scale and for qualitative feedback, while others ask for qualitative feedback on a series of questions about such things as academic contribution, knowledge of the literature, quality of the methodology (if relevant), quality of findings, and quality of writing.
Overall, PRs are typically asked to report on the appropriateness of the methods and statistical analyses that have been used, the thoroughness of the literature review, the quality of the writing, and the flow of arguments in the paper. Therefore, the quality of the English is not something PRs are generally required to report on; however, if they feel there is an issue with the English, they are expected to comment on it.
After receiving the feedback from all the PRs, the journal editor decides about the next step for the manuscript, and they often send a copy of all of the PRs’ comments to the involved PRs by email. This is very helpful and interesting for the PRs because they are able to compare their review with those of the other PRs.
Do peer reviewers follow the progress of the papers after their initial review?
In many cases, the PRs receive a reply from the editor to thank them for the review, and the PRs get to hear about the paper again if it is sent for second (third, fourth) revisions and re-submissions because it is generally the practice to ask the original PRs to comment on the revised versions wherever possible. Once the paper is accepted, the journal editors often write to inform the PR of the outcome and include the comments from the other PRs. However, in other cases, the journal editor may not advise the PRs of the final decision. For example, Sara only found out that one of the papers she reviewed was published coincidentally “because a friend happened to point it out to me (because it was relevant to my work).”