TB: Advice for Writing Peer Reviews (Sutton)

Précis

Rich Sutton gives a short, prescriptive structure for a peer review: introduction, decision, argument, denouement, details. The animating premise is that the author is the one doing the real work and that the review's job is to give honest analysis without disrespecting the author — and to keep "what the paper must do to be acceptable" cleanly separate from "what would make the paper better in the reviewer's opinion."

Key Takeaways

The author is king

"The author is the one doing the real work and the success of any meeting or journal depends on attracting good authors. So be respectful to them, while giving your best analysis."

Five-section structure

Introduction — Summarize the paper in a few sentences. Be neutral. Include the perspective from which the work might be a good paper. Audience: editor + author; demonstrates that the reviewer understood the work.
Decision — Overall assessment in a few sentences. Clear accept/reject recommendation, with the reason in general terms (e.g. "flaws in the experimental design make it impossible to assess the new ideas"). "The bottom line is: does this paper make a contribution?" The editor should be able to write the summary letter from this section alone if reviewers agree.
Argument — Substance that backs up the decision. Detailed critique, defended with examples. Audience: author + other reviewers. Editor reads this only if reviewers disagree.
Denouement — Suggestions for improving the paper. "It is important that these are suggestions, advice to the author, not reasons for the decision described above." The decision is over; this section is purely constructive.
Details — Typos, unclear sentences, citation gaps. If you flag missed literature, "provide a full citation to the work."

Required-vs-preferred separation

For a rejection, distinguish cleanly between (a) changes required for the paper to make a contribution and (b) changes that would just make the paper better in the reviewer's opinion. "Authors hate it when a reviewer seems to reject because the paper was not written the reviewer's way."

Why this matters / Connections

John's annotation: "Very good advice for reviewing papers and organizing the summary." The structural lesson — separate the decision from the rationale, separate the rationale from the wishlist — generalizes outside peer review to any critical feedback. The denouement-as-strictly-constructive rule is the same discipline as "write code review comments as either blocking or optional, never blended."

Source

Advice for writing peer reviews — Rich Sutton, IncompleteIdeas.net