A quick guide to reviewing papers quick
Let me start by saying reviewing papers is a nigh-sacred duty that we should all treat with respect. The gold standard for scientific knowledge generation is peer review; if we skimp on our duties as reviewers, we cheapen the value of the work we all do. So “reviewing quick” should be a tertiary goal; the primary goal should be fairly and compassionately assessing the work. The secondary goal should be producing feedback that helps authors’ improve their work.
That said, there are many times where I find myself having to review a lot of papers in a short amount of time [1]. Doing so has complicated my perspective on what it means to be a “good reviewer” and necessitated my figuring out how to write “good reviews” without it cannibalizing all of my time.
I think there’s a misconception among many folks that reviewing necessitates carefully poring through and scrutinizing every word and every sentence. This strategy has several drawbacks:
- It is time-consuming
- It encourages nit-picky feedback that misses the forest for the trees
- It dilutes author attention (how many times have you heeded every detail of a super long review?)
- It privileges presentation over substance (better writing is a good thing, of course, but the writing is only a medium to communicate the work)
In short, the “carefully scrutinizing every word in a slog to the end”-approach to reviewing takes a lot of reviewer time and risks the production of low-impact feedback that can frustrate and disillusion authors.
Here’s an alternative approach that I’ve found to be more time-efficient, maintain high quality, and make reviewing more enjoyable [2].
First, read through the abstract and the introduction of a paper carefully. The abstract and the introduction, if written well, should clearly articulate what work was done, why it was done, and how it was done [3]. Given this information, take a moment to generate a list of questions that you would need answered to feel comfortable accepting the paper. I like to focus on the following criteria in generating these questions [4], but please note that the criteria I’m highlighting here are a reflection of my epistemological background and may not be appropriate for all types of work:
- Generalizability: Does the knowledge generated by this paper generalize beyond the scope of this work? You might ask questions about the representativeness of a study sample or dataset here, for example. Note that not all work needs to focus on generalizability; some valid contributions are exploratory. Use your judgment about this based on the type of contribution the paper claims to be making.
- Replicability: Is there enough information in the paper for independent researchers to replicate the methods and validate the results? You might ask about study instruments here, methodological details, or the use of proprietary data.
- Validity: Are the claims made in the paper supported by the data and evidence provided? You might question the stated findings here, or point out missing related literature that complicates the reported finding. If pointing missing literature, make sure to cite the literature that is missing.
- Novelty: Are the claims made in the paper new or unique? Does the paper appropriately engage with prior relevant work? You might ask if the findings are appropriately situated in the literature to differentiate what is known vis-a-vis what is claimed as a contribution in the paper.
- Ethics: Was care taken to mitigate the potential harms of this work? Does the work have the potential to cause harm in society? You might ask user consent, about how data was acquired, about strategies to mitigate harmful applications of the stated contribution, or about how the paper might disproportionately harm some members of society more than others.
- Impact: What does this paper add to our knowledge that we do not already know? How important is it? This is the “fuzziest” category, so I encourage you to be open-minded about what constitutes impact. You might ask if and how the research benefits the research community or society; Does it bridge two disparate perspectives? Will it contribute towards a more equitable world? Does it measure a commonly held assumption?
Note that you don’t need to generate corresponding questions for each of these criteria for every paper you review. In fact, if you feel uncomfortable about your expertise in assessing, e.g., the validity or replicability of a particular paper, perhaps you should focus on criteria that you are more well-suited to assess [5]. You can also reach out to the person who assigned you the review and ask them what they were hoping to gain by having your perspective.
Regardless, as you read through the abstract and intro, think about these categories and come up with a minimum set of questions that, if answered, would make you feel happy about accepting the paper.
Using those questions as a guide, read the rest of the paper to find the answers. You don’t need to pore through every word; just try to answer your own questions.
Sometimes, as you look for the answers to your own questions, you may find other issues with the paper. You should ask yourself how important this ad-hoc issue is — remember, no paper is perfect, and your job is not to spot every imperfection. If the ad-hoc issue is critically important (e.g., if it is the difference between an accept or reject), then add it to your review. If it is not critically important, then ask yourself if it is important enough and how to articulate in a way that does not detract from more important points. Just remember, the more you add to your review, the more diluted the message is likely to get.
By the time you have finished answering your questions (or failed to find the answers to them), you should have a skeleton for what you will ultimately write in your review. If you have a question that was unanswered or inadequately answered — well, perhaps that is a request for a minor or major revision (assuming all else is okay).
I also recommend writing the review as you are reading the paper. I understand some people like to read papers first, take notes, and assemble a coherent review later — this is a fine strategy, and one that I had used myself for many years, but I find it is much slower. Moreover, writing is thinking — by forcing yourself to summarize a paper’s contribution as you are reading it, you are engaging with the content more deeply as you are reading which can, in turn, help you elicit the appropriate questions to ask in your review.
Using this approach, I’ve reduced the amount of time I need to spend on each review considerably while also, I think, increasing the impact of my reviews in decision-making and driving author revisions.
—
Footnotes:
[1] I am better at requesting reasonable reviewing loads now, but there were times earlier in my faculty career where I would serve on multiple big program committees at once: (e.g., CHI, IMWUT, and USENIX Security).
[2] Note that it is just *an* alternative. There are many different ways to review; I make no claim that this is the best one, nor that I am the best reviewer. But, I have received at least 15 recognitions for good or excellent reviews across CHI, IMWUT, USENIX Security, SOUPS, and UIST so I think I am, at least, not a bad reviewer.
[3] If the abstract and intro fail to establish these points, chances are the paper is not ready for publication, but this may vary by discipline. In HCI and Security, I find this to be generally true.
[4] Note that there are other criteria too — e.g., fit. But these are the ones I consider most important. I don’t care about “fit” with a venue nearly as much; I let the authors figure that out.
[5] Though, if you have no ability to assess the validity of a submission, you should reach out to the person who assigned you the review and let them know; chances are, you shouldn’t be a reviewer for that submission.
