A quick guide to reviewing papers quick

  • It is time-consuming
  • It encourages nit-picky feedback that misses the forest for the trees
  • It dilutes author attention (how many times have you heeded every detail of a super long review?)
  • It privileges presentation over substance (better writing is a good thing, of course, but the writing is only a medium to communicate the work)
  • Generalizability: Does the knowledge generated by this paper generalize beyond the scope of this work? Or is it a case study? You might ask questions about the representativeness of a study sample or dataset here, for example. Note that not all work needs to focus on generalizability; some valid contributions are exploratory. Use your judgment about this based on the type of contribution the paper claims to be making.
  • Replicability: Is there enough information in the paper for independent researchers to replicate the methods and validate the results? You might ask about study instruments here, methodological details, or the use of proprietary data.
  • Validity: Are the claims made in the paper supported by the data and evidence provided? Are the claims made in the paper new or unique? Does the paper appropriately engage with prior relevant work? You might question the stated findings here, or point out missing related literature that complicates the reported finding. If pointing missing literature, make sure to cite the literature that is missing.
  • Ethics: Was care taken to mitigate the potential harms of this work? Does the work have the potential to cause harm in society? You might ask user consent, about how data was acquired, about strategies to mitigate harmful applications of the stated contribution, or about how the paper might disproportionately harm some members of society more than others.
  • Impact: What does this paper add to our knowledge that we do not already know? How important is it? This is the “fuzziest” category, so I encourage you not to disqualify a paper based on your subjective perception of impact. However, you might ask if and how the research benefits the research community or society; does it bridge two disparate perspectives? Will it contribute towards a more equitable and just world?

Footnotes:

  1. Follow me on Twitter (spam frequency: daily — weeklyish)
  2. Subscribe to my YouTube channel (spam frequency: ¯\_(ツ)_/¯)
  3. Subscribe to my mailing list (spam frequency: In Theory? Once a month with links to whatever I’ve written. In Practice? Once a year or so?)

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sauvik Das

Sauvik Das

Assistant Professor of Human-Computer Interaction at Carnegie Mellon University. Formerly at Georgia Tech. Ph.D. from CMU HCII. HCI, Security, Data Science.