After further review

Abstract

In the two years that I have been Editor in Chief of the JCI, the editors and I have sent approximately 2,500 papers for external review. The referee reports that we’ve received have spanned the spectrum from incredibly insightful to completely unhelpful. Together with a longer piece immediately following this editorial, I would like to reflect on what I think makes a good review.

Every spring semester two of my colleagues and I teach a journal club course to the immunology graduate students at the University of Pennsylvania, in which the students present and discuss a selection of recently published papers. As part of the course, we also try to teach the students how to review papers. We choose a few articles from top-tier journals and ask them to serve as referees for those papers as if they had been submitted to that particular journal for publication. We provide them examples of reviews that we have written and spend some time talking about what makes a review good (or bad) and what information editors and authors need. As I like to think that my role as JCI Editor in Chief provides me a particularly good perspective, I have given some thought as to what that perspective has actually shown me.

At the JCI, we almost always seek three outside referees, and in general, I try to read all of their reviews, meaning that I have read close to 7,500 reviews in the past two years. I have had many surprises as editor, one being that it is uncommon to have all three reviewers agree that a given paper is good or bad. My estimate is that the rate of reviewer concordance is only about 10%–20%. The editors take their job seriously, so our decisions are not based on simple two-to-one majorities but on our own reading of the paper and the strengths of the reviewers’ arguments. This of course has gotten me thinking again about the key elements of good reviews, and the fact that if you, as a referee, would like to persuade the editors, it may be helpful to keep a few things in mind:

1. Be dispassionate. Strident reviews aggravate authors and make editors wonder whether you have ulterior motives. Remember Kant’s categorical imperative — “Act only according to that maxim by which you can at the same time will that it would become a universal law” (i.e., do unto others only as you would have them do unto you) — and imagine how you would feel as an author if you received the comments you were writing as a reviewer. If you just can’t resist, at least confine your wrath to the “confidential comments to editors” section.

2. Provide references to back up your points. A statement such as “the findings that serum porcelain predisposes to asthma are not novel” may be questioned by the authors and by the editors as well. Unless your assertion is patently obvious (in which case, hopefully we wouldn’t have sent the manuscript for review in the first place), providing a citation will greatly help us.

3. In the confidential comments to editors section, please tell us whether or not the paper, even if revised, is suitable for the JCI and what priority it would have. I really can’t overemphasize this point. We ask these questions on our review form, but they frequently go unanswered. Telling us what is wrong with a paper and providing suggestions to the authors is not the same as letting us know whether it has a high enough priority for JCI.

4. Be passionate about papers you like. I know this goes against point number 1, but I have never yet seen a complaint about a reviewer who liked a paper too much. A single enthusiastic, persuasive reviewer can sometimes overcome two negative reviewers and even skeptical editors. This is true especially if you are able to speak to why you like the study, what paradigm it overturns, how it advances the field or could change clinical treatment, etc.

5. Don’t trash the paper to the editors in the confidential comments and say only wonderful things to the authors. This occurs often and only confuses authors as to why a paper may be rejected.

6. Don’t shift the goalposts. On a review of an invited revision, do not raise new issues, unless they occur as a result of new data added in response to the original review. Nothing aggravates authors more than this.

7. Don’t volunteer to write a Commentary. You will only be seen as self-serving.

8. If you agree to review a paper, review it. Shirking responsibility happens much more often than I would have thought. We will understand if you are a few days late, but it is harder to understand not providing a review at all. In these cases, we have to seek alternate reviews, which usually takes an additional two weeks. Consider how you feel as an author when decisions are held up because of tardy, or absent, referees.

To end on a positive note, the vast majority of the reviews we receive are extremely thoughtful, well reasoned, and provide useful advice. For that, we are grateful, and I wish to take this opportunity to thank all of you who review frequently and carefully for us. Your services are invaluable. Although they can’t thank you directly, the authors also are grateful, as good and speedy reviews help us make rapid and informed decisions.