Reviewing papers can be daunting, especially when doing it for the first time. Beyond giving back to the research community we expect constructive reviews from, there is a tangible advantage for your own work. Writing reviews helps you understand what reviewers are looking for. If you have ever been annoyed that the reviewers do not understand what you have written, reviewing yourself is the perfect medicine. It makes you aware of how hard it is to clearly and concisely communicate research findings.
Types of reviews
Painting with a broad brush, there are two types of review procedures:
The traditional is where the authors do not “converse” with the reviewers. You get the reviews, and that’s it. There is no means to make amends but to resubmit (to a different venue)—increasing time-to-publish considerably. If you are lucky and get an “accept with minor revisions”, you can change the paper. But in that case, the reviewers might not even need to read the updated version. Journals often employ this strategy.
The rebuttal is where authors can interact with the reviewers. It has a larger overhead and time commitment but is more gracious for unclear claims and errors. It also embodies the cooperative spirit of science, where everyone tries to push the frontiers of knowledge forward. At least, in the best case1. AI conferences often use openreview.net, which is akin to a forum for papers. Reviewers comment on the papers and the authors respond, working out a better version.
Before you begin: your ignorance
Be clear about your ignorance. If you are not an expert in the field and cannot ask for help from someone, state this prominently in your review. Some venues require you to set a confidence score for this purpose.
If you are unsure, play the steel man, not the straw man card. Assume you might have misunderstood the authors’ claim—but still point this out so the authors can clarify.
The clarity threshold
The first thing people often check is clarity. That is, if the paper is very hard to follow, that’s a rejection—or if there is a rebuttal, there might be a chance to save it this time.
Note that clarity is not about the science itself but its presentation.
This was hard to accept for me, as I wanted to dig into the details, assuming that I must be an idiot that it takes me so long to understand the claims.
I did not realize that as a reviewer, you are a filter for the reader. You want to ensure the reader can easily follow the paper’s arguments.
This does not mean that a six-year-old should understand everything. But at least the main idea should be clear to most readers.
A quick reminder from my previous post: the following items are a decent proxy for clarity and readability:
a conceptual “Figure 1” visualizing the main contributions
an intuition for the proposed method
an algorithm (in pseudocode)
text highlights and color-coding (used sparingly)
Things that lower clarity:
math for math's sake
very long sentences
exotic words when there is a simpler alternative
The quality threshold
Clarity is worthless without correctness. Proofs need to be sound2, and the experiments reproducible.
The novelty threshold
Is the work novel? Often, reviewers have very different expectations of what novelty means. I recommend reading Michael J. Black's thoughts on the topic.
The ethical threshold
Are the authors acknowledging relevant research? Especially in fast-moving fields, it can be tricky to keep up with everything. Is there any ethical issue (e.g., if human subjects are involved)?
Your review should also adhere to ethical rules. For example, even if you have relevant papers the authors have not cited, don’t engage in “cite me to get published” battles. It’s OK to suggest papers they have overlooked—including, but not exclusively, your own.
Conclusion
You should write reviews you would be happy to accept. That is, you should be constructive, respectful, and transparent. If you decide not to change your mind, provide concrete reasons why you do so. Go the extra mile to suggest improvements even if you reject the paper. Also, acknowledge the effort the authors put into the paper and the rebuttal, and acknowledge receiving their response. Remember,
reviewing is not a battle of egos, but one for the truth.
Be kinder than necessary, and even if you reject the paper, write down what it would take to accept it.
Writing reviews is free work, sometimes assigned to inexperienced graduate students. But sometimes, the reviewers are not respectful. This is where the memes about “Reviewer 2” come from
A.k.a., the academic version of the soundcheck