Reviewer two became the epitome of the peer review process. Granted, reviews can be superficial, aggressive, and downright false. It is free work, sometimes trusted onto inexperienced students. The cynical argument even says reviewers aim for rejection to keep their field more exclusive. Still, peer review ought to improve Science.
Venting about useless reviews and low scores can feel good, but that's not how good Science is made or how papers get accepted. Thus, I will turn to what we can do. Let's start with rebuttal 101:
The role of the rebuttal is to convince the reviewers ("the judges") that your science is good and worthy of publication.
It is like a courtroom situation.
The rebuttal begins when you hit "submit"
Submitting your work provides a sense of accomplishment and frees your mind (you are taking some time off, aren't you?). It would be a mistake, though, to think that your work is done.
You should think about what could go wrong to prepare for the rebuttal.
The Romans called this premeditatio malorum. It is not a splurge of pessimism but a rational assessment of your own work's shortcomings and extensions. You know the missing pieces and weak spots best (see the questions at the end to help identify these).
Plan
Conferences specify the reviews' release date, so block that time ahead. If you need additional experiments, you can figure out new software, collect data, and design the questionnaire.
Write reviews yourself!
Thinking with the reviewer's head helps determine what they will ask for. If you do, consider this checklist I wrote to provide a starting point.
Emotion check
You can be the epitome of stoicism and still get angry, sad, and devastated when a review feels like a personal attack. In that case, sleep on it. What matters is doing good Science, not that your ego emerges unbruised. The intellectual humility to value outcome over ego is invaluable but takes time to learn.
The reviewers are right, you are wrong
Leading an ad hominem counterattack will not get you a better score. Hanlon's razor suggests that:
don't attribute to malice what can be explained by stupidity
Substitute stupidity with time pressure, lack of experience, or a lousy reviewer match.
If the reviewer did not understand something, you did not explain it well enough. Writing clearly involves guessing the background knowledge of a technical audience, which is very hard! Remember: many reviewers care about what they are doing (especially for smaller, thematic conferences). Even if they aren't, it won't help you. Again, the goal is to convince the reviewers. Doing good Science comes first, but that's before hitting submit.
Get an overview of the reviews
Let's get down to brass tacks. I found it very helpful to keep a shared document with all the reviews, where you can
organize common themes
color-code strengths, weaknesses, additional experiments, etc.
keep a checklist
distribute tasks
I found this document also full of helpful advice. This document will form the basis of your response and the ammunition to reason back.
Handling (ir)rationality
You still might find irrational/false statements in the review. The problem there is that
You can't reason someone out of a notion that they didn't reason themselves into.—Kevin Kelly
If it's a violation of ethics or downright false, you might consider contacting the area chair (use it sparingly).
Diplomatic disagreement
You can disagree. Even if a reviewer attacks you, don't attack back, use reason (remember, we ought to do good Science). But you need to have rock-solid evidence.
Divide et impera
The most ingenious advice I have heard, and perhaps the greatest benefit of a shared document summarizing all reviews, is that you can use a reviewer's praise to disarm another reviewer's criticism.
The power of the joint response
Some venues allow you to write a separate joint response to all reviewers. This way, you can address common themes. The joint response is also for the area chair (AC), i.e., the more senior person handling a batch of submissions. They won't have time to read through everything, so a joint response (highlighting the strengths of your work in the reviewers' words) can convince the AC that your work is worthy of acceptance.
Mindset
Strive to do great Science. Even then, some of your papers will be rejected. It is a hard lesson to learn, but you need to detach the quality of your Science from your paper being accepted. But check your ego to avoid using this as an excuse to address the limitations of your work.
If you are interested in a real-world study of review inconsistency, check out the NeurIPS consistency experiment.
Asking the right questions
To best prepare for the rebuttal, you need to ask good questions about the weak spots of your own paper. As you have been immersed in the topic to the deepest, you have a head start (barring the last point, for that, ask for feedback from others). Use this as an advantage.
If you would review your own paper, what would you ask?
What are the weak points of your paper?
Are there additional experiments that would add value (but maybe you didn't have time to run them before the submission deadline)?
Is there a theoretical question/result you expect reviewers to ask for?
Is there additional analysis or a new presentation of results that could improve clarity or novelty?
Is there a gap in the literature review?
Are there unstated assumptions/facts that seem obvious to you (i.e., the curse of knowledge)?
