A quick guide to reviewing papers quick

Let me start by saying reviewing papers is a nigh-sacred duty that we should all treat with respect. The gold standard for scientific knowledge generation is peer review; if we skimp on our duties as reviewers, we cheapen the value of the work we all do. So “reviewing quick” should be a tertiary goal; the primary goal should be fairly and compassionately assessing the work. Emphasis on compassionately. The secondary goal should be producing feedback that helps authors’ improve their work.

That said, last year, I reviewed a lot [1]. Doing so complicated my perspective on what it means to be a “good reviewer” and necessitated my figuring out how to write “good reviews” without it cannibalizing all of my time.

I think there’s a misconception among many graduate students, at least, that reviewing necessitates carefully poring through and scrutinizing every word and every sentence. This strategy has several drawbacks:

  • It is time-consuming

In short, the “carefully scrutinizing every word in a slog to the end”-approach to reviewing takes a lot of reviewer time and risks the production of low-impact feedback that can frustrate authors.

Here’s an alternative approach that I’ve found to be more time-efficient, maintain high quality, and make reviewing more enjoyable [2]. This approach should work, I think, for most HCI and HCI-adjacent papers. It may work more broadly, but I cannot guarantee that.

First, read through the abstract and the introduction of a paper carefully. The abstract and the introduction, if written well, should clearly articulate what work was done, why it was done, and how it was done [3]. Given this information, take a moment to generate a list of questions that you would need answered to feel comfortable accepting the paper. I like to focus on the following criteria in generating these questions [4]:

  • Generalizability: Does the knowledge generated by this paper generalize beyond the scope of this work? Or is it a case study? You might ask questions about the representativeness of a study sample or dataset here, for example. Note that not all work needs to focus on generalizability; some valid contributions are exploratory. Use your judgment about this based on the type of contribution the paper claims to be making.

Note that you don’t need to generate corresponding questions for each of these criteria for every paper you review. In fact, if you feel uncomfortable about your expertise in assessing, e.g., the validity or replicability of a particular paper, perhaps you should focus on criteria that you are more well-suited to assess [5]. You can also reach out to the person who assigned you the review and ask them what they were hoping to gain by having your perspective.

Regardless, as you read through the abstract and intro, think about these categories and come up with a minimum set of questions that, if answered, would make you feel happy about accepting the paper.

Using those questions as a guide, read the rest of the paper to find the answers. You don’t need to pore through every word; just try to answer your own questions.

Sometimes, as you look for the answers to your own questions, you may find other issues with the paper. You should ask yourself how important this ad-hoc issue is — remember, no paper is perfect, and your job is not to spot every imperfection. If the ad-hoc issue is critically important (e.g., if it is the difference between an accept or reject), then add it to your review. If it is not critically important, then ask yourself if it is important enough and how to articulate in a way that does not detract from more important points. Just remember, the more you add to your review, the more diluted the message is likely to get.

By the time you have finished answering your questions (or failed to find the answers to them), you should have a skeleton for what you will ultimately write in your review. If you have a question that was unanswered or inadequately answered — well, perhaps that is a request for a minor or major revision (assuming all else is okay).

Using this approach, I’ve reduced the amount of time I need to spend on each review considerably while also, I think, increasing the impact of my reviews in decision making and driving author revisions.

I would also be curious to hear how others have managed to make their reviewing processes more time efficient without reneging on the core duty.

Footnotes:

[1] I served on three program committees: CHI, IMWUT, and USENIX Security. Each of these is a non-trivial load in and of itself, but all three at once? Oh, also I was on two faculty hiring committees. Anyway.

[2] Note that it just *an* alternative. There are many different ways to review; I make no claim that this is the best one. Should you listen to me about reviewing? Honestly, no one should listen to me about anything. But I have gotten a bunch of “excellent review” designations for my reviews, so I think I am, at least, not a bad reviewer.

[3] If the abstract and intro fail to establish these points, chances are the paper is not ready for publication.

[4] Note that there are other criteria too — e.g., fit. But the above five are the ones I consider most important. I don’t care about “fit” nearly as much; I let the authors figure that out.

[5] Though, if you have no ability to assess the validity of a submission, you should reach out to the person who assigned you the review and let them know; chances are, you shouldn’t be a reviewer for that submission.

Thanks for reading. If you read this and thought: “whoah, definitely want to be spammed by that guy”, there are three ways to do it:

  1. Follow me on Twitter (spam frequency: daily — weeklyish)

You also don’t have to do any of those things, and we will both be fine.

Assistant Professor of Interactive Computing at Georgia Tech. Ph.D. from CMU HCII. HCI, Security, Data Science.