Unprofessional review comments affect authors from underrepresented groups disproportionately. Journal editors must take a more active role to minimize the potentially adverse effects of such comments.
Just over a decade ago, as an Associate Editor of the journal Nature Geoscience, I was confronted with comments from a reviewer that focused more on the authors of the paper than their work. Moreover, the reviewer’s scarcely concealed preference for a model was clearly inhibiting an impartial assessment. As the reviewer did not heed my request for a modified, more professional review, I was left with no alternative but to disregard it. Had I passed on the review to the authors as is, I would have shirked my responsibility as a gatekeeper.
Unfortunately, it seems that the gate is not being guarded as zealously as it should be. Recently published analysis of an anonymous survey reveals that unprofessional review comments are pervasive. More importantly, the study suggests that such comments are likely to have an especially adverse effect on the productivity and career advancement of those who are traditionally underrepresented because of their gender and/or race or ethnicity.
The survey, which was international in scope, focused on those from the science, technology, engineering and mathematics (STEM) community. It sought to gauge both the pervasiveness of unprofessional reviews and the perceived effects of such reviews on such aspects as scientific aptitude, productivity and career advancement of the recipients. For the purpose of the survey, reviews were classified as unprofessional if they contained irrelevant or offensive comments, and not ones that were frank and even harsh but nevertheless constructive. Survey respondents were asked to provide demographic information that revealed their gender and racial/ethnic identities, which helped evaluate the effects of unprofessional reviews on individuals belonging to different groups.
The study found that almost 60% of the respondents received unprofessional reviews that included such comments as “This paper is, simply, manure.” Those from underrepresented groups were no more likely to receive such reviews than those from well represented groups. Yet, individuals from the former groups were more likely to report increased self-doubt and delayed academic productivity following the receipt of unprofessional reviews. The likelihood was the highest among women of colour and non-binary people of colour. Although this study did not identify how exactly unprofessional reviews might lead to adverse consequences, it suggests that such reviews might serve to reinforce well known disadvantages faced by underrepresented groups. For example, the glass ceiling and workplace harassment for women and barriers to recruitment for racial and ethnic minorities.
“A good editor should be able to and willing to deal with unprofessional reviews skilfully but firmly.”
Irrespective of the extent and degree of their longer-term consequences, unprofessional reviews are clearly not acceptable, even from unpaid reviewers who may be arguably overworked. The study’s authors recommend a set of actions to weed such reviews out, ranging from institutionalizing peer-review mentorship to devising clear guidelines and exercising greater editorial oversight. There is also an indication from this study that open peer review might help, because only a very small percentage of respondents reported receiving unprofessional comments from journals that institute open peer review. In that context, a study published in Nature Communications last year suggests that open peer review did not “compromise the inner workings of the peer review system”. However, the openness to this type of review varies by discipline and some communities are still quite resistant to the notion.
As a former journal editor, I have no doubt that editorial oversight is crucial, because it does not rely on voluntary adherence to codes on part of reviewers. A good editor should be able to and willing to deal with unprofessional reviews skilfully but firmly. But this needs time, which is unfortunately at a premium for most editors who are practising academics, not fulltime professionals. That they nevertheless manage to do a stellar job is testimony to their commitment.
Clear understanding and acceptance of the role and responsibility of a journal editor are also needed. I could do what I did at least in part because there was never any doubt that decision-making was my responsibility and not something to be outsourced to reviewers. That this is not the case everywhere is clear when I participate in the publishing ecosystem as a reviewer. I have often found myself questioning an editor’s decision to rely on my judgement instead of their own.
Minimizing unprofessional reviews will have only a small role to play in the overall effort to level the playing field for underrepresented groups. Nevertheless, doing so is much easier than tackling other pernicious but resistant biases, which is a fine reason to act now in view of this study’s findings.
Ninad is an alumni of Elevate Scientific, having been its Managing Editor between 2016-2020.