A quick guide to reviewing papers quick

Let me start by saying reviewing papers is a nigh-sacred duty that we should all treat with respect. The gold standard for scientific knowledge generation is peer review; if we skimp on our duties as reviewers, we cheapen the value of the work we all do. So “reviewing quick” should be a tertiary goal; the primary goal should be fairly and compassionately assessing the work. Emphasis on compassionately. The secondary goal should be producing feedback that helps authors’ improve their work.

That said, last year, I reviewed a lot [1]. Doing so complicated my perspective on what it means to be a “good reviewer” and necessitated my figuring out how to write “good reviews” without it cannibalizing all of my time.

I think there’s a misconception among many graduate students, at least, that reviewing necessitates carefully poring through and scrutinizing every word and every sentence. This strategy has several drawbacks:

• It is time-consuming
• It encourages nit-picky feedback that misses the forest for the trees
• It dilutes author attention (how many times have you heeded every detail of a super long review?)
• It privileges presentation over substance (better writing is a good thing, of course, but the writing is only a medium to communicate the work)

In short, the “carefully scrutinizing every word in a slog to the end”-approach to reviewing takes a lot of reviewer time and risks the production of low-impact feedback that can frustrate authors.

Here’s an alternative approach that I’ve found to be more time-efficient, maintain high quality, and make reviewing more enjoyable [2]. This approach should work, I think, for most HCI and HCI-adjacent papers. It may work more broadly, but I cannot guarantee that.

First, read through the abstract and the introduction of a paper carefully. The abstract and the introduction, if written well, should clearly articulate what work was done, why it was done, and how it was done [3]. Given this information, take a moment to generate a list of questions that you would need answered to feel comfortable accepting the paper. I like to focus on the following criteria in generating these questions [4], but please note that the criteria I’m highlighting here are a reflection of my epistemological background and may not be appropriate for all types of work:

• Generalizability: Does the knowledge generated by this paper generalize beyond the scope of this work? Or is it a case study? You might ask questions about the representativeness of a study sample or dataset here, for example. Note that not all work needs to focus on generalizability; some valid contributions are exploratory. Use your judgment about this based on the type of contribution the paper claims to be making.
• Replicability: Is there enough information in the paper for independent researchers to replicate the methods and validate the results? You might ask about study instruments here, methodological details, or the use of proprietary data.
• Validity: Are the claims made in the paper supported by the data and evidence provided? Are the claims made in the paper new or unique? Does the paper appropriately engage with prior relevant work? You might question the stated findings here, or point out missing related literature that complicates the reported finding. If pointing missing literature, make sure to cite the literature that is missing.
• Ethics: Was care taken to mitigate the potential harms of this work? Does the work have the potential to cause harm in society? You might ask user consent, about how data was acquired, about strategies to mitigate harmful applications of the stated contribution, or about how the paper might disproportionately harm some members of society more than others.
• Impact: What does this paper add to our knowledge that we do not already know? How important is it? This is the “fuzziest” category, so I encourage you not to disqualify a paper based on your subjective perception of impact. However, you might ask if and how the research benefits the research community or society; does it bridge two disparate perspectives? Will it contribute towards a more equitable and just world?

Note that you don’t need to generate corresponding questions for each of these criteria for every paper you review. In fact, if you feel uncomfortable about your expertise in assessing, e.g., the validity or replicability of a particular paper, perhaps you should focus on criteria that you are more well-suited to assess [5]. You can also reach out to the person who assigned you the review and ask them what they were hoping to gain by having your perspective.

Regardless, as you read through the abstract and intro, think about these categories and come up with a minimum set of questions that, if answered, would make you feel happy about accepting the paper.

Using those questions as a guide, read the rest of the paper to find the answers. You don’t need to pore through every word; just try to answer your own questions.

Sometimes, as you look for the answers to your own questions, you may find other issues with the paper. You should ask yourself how important this ad-hoc issue is — remember, no paper is perfect, and your job is not to spot every imperfection. If the ad-hoc issue is critically important (e.g., if it is the difference between an accept or reject), then add it to your review. If it is not critically important, then ask yourself if it is important enough and how to articulate in a way that does not detract from more important points. Just remember, the more you add to your review, the more diluted the message is likely to get.

By the time you have finished answering your questions (or failed to find the answers to them), you should have a skeleton for what you will ultimately write in your review. If you have a question that was unanswered or inadequately answered — well, perhaps that is a request for a minor or major revision (assuming all else is okay).

Using this approach, I’ve reduced the amount of time I need to spend on each review considerably while also, I think, increasing the impact of my reviews in decision making and driving author revisions.

I would also be curious to hear how others have managed to make their reviewing processes more time efficient without reneging on the core duty.

—

Footnotes:

[1] I served on three program committees: CHI, IMWUT, and USENIX Security. Each of these is a non-trivial load in and of itself, but all three at once? Oh, also I was on two faculty hiring committees. Anyway.

[2] Note that it just *an* alternative. There are many different ways to review; I make no claim that this is the best one. Should you listen to me about reviewing? Honestly, no one should listen to me about anything. But I have gotten a bunch of “excellent review” designations for my reviews, so I think I am, at least, not a bad reviewer.

[3] If the abstract and intro fail to establish these points, chances are the paper is not ready for publication.

[4] Note that there are other criteria too — e.g., fit. But the above five are the ones I consider most important. I don’t care about “fit” nearly as much; I let the authors figure that out.

[5] Though, if you have no ability to assess the validity of a submission, you should reach out to the person who assigned you the review and let them know; chances are, you shouldn’t be a reviewer for that submission.

Thanks for reading. If you read this and thought: “whoah, definitely want to be spammed by that guy”, there are three ways to do it:

1. Follow me on Twitter (spam frequency: daily — weeklyish)
2. Subscribe to my YouTube channel (spam frequency: ¯\_(ツ)_/¯)
3. Subscribe to my mailing list (spam frequency: In Theory? Once a month with links to whatever I’ve written. In Practice? Once a year or so?)

You also don’t have to do any of those things, and we will both be fine.