TB: The Rise and Fall of Peer Review (Mastroianni)

Core Thesis

Peer review was the largest-ever experiment in scientific quality control — and it failed. Science is a strong-link problem (progress depends on the best work), not a weak-link problem, so stamping out bad ideas at publication is both costly and misguided.

Key Takeaways

The experiment

  • Peer review began after World War II as a government accountability mechanism: funders convened reviewers to avoid wasting money on foolish proposals.
  • It became a massive, expensive intervention with universal scope — every scientist on Earth was enrolled; scientists spend ~15,000 researcher-years reviewing papers annually.
  • "Huge interventions should have huge effects." If peer review improved science, that should be obvious. It isn't.
  • Nobel Prize-winning pre-WWII discoveries are rated "just as good or even better" than post-peer-review work.

Evidence it doesn't work

  • Studies with deliberate errors seeded into papers show reviewers catch only 25–30% of major flaws (three separate studies converge: ~25%, ~29%, ~30%).
  • Fraudulent papers get published routinely — knock-down real-world evidence of failure. Andrew Wakefield's debunked vaccine-autism paper stayed in The Lancet 12 years.
  • A recent paper used a literal capital letter "T" instead of actual error bars and passed review.
  • ~20% of genetics papers contain useless data due to Excel autocorrecting gene names to dates.
  • ~50% of authors retracted submissions when asked to provide raw data.
  • Reviews are shredded after publication; most journals don't publish them; nobody asks what reviewers said — suggesting nobody believed the reviews mattered.
  • Scientists already take unreviewed work seriously without thinking twice (preprints, conference talks, datasets).

Historical example: Einstein

  • Einstein submitted only one paper to peer review — was upset by the rejection, published elsewhere, and never used peer review again. Most of his major papers were not peer reviewed.

The core conceptual error

  • Science was treated as a weak-link problem: progress limited by the worst work; stamp out bad ideas at all costs; acceptable to whack good ideas in the process.
  • But science is a strong-link problem: progress depends on the quality of the best work.
  • Under strong-link logic, burying a few good ideas in the process of blocking bad ones is actively harmful, not acceptable collateral damage.
  • Geocentrist reviewers would have rejected heliocentrism; eugenics-era reviewers would have blocked papers showing racial equality — peer review institutionalizes paradigm suppression.

The alternative

  • Publish work directly, write in accessible language with personality, share all data and code openly.
  • Let community evaluate without formal gatekeeping; trust time and utility to sort good ideas from bad.
  • Mastroianni's own unreviewed paper got more views, more read-to-completion, more thoughtful unsolicited reviews, and an NPR interview than his PNAS peer-reviewed paper.

Key Quotes

  • "The results are in. It failed."
  • "Huge interventions should have huge effects. If peer review improved science, that should be pretty obvious, and we should be pretty upset and embarrassed if it didn't."
  • "All we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal."
  • "Once a paper gets published, we shred the reviews… Nobody cares to find out what the reviewers said or how the authors edited their paper in response, which suggests that nobody thinks the reviews actually mattered in the first place."
  • "progress depends on the quality of our best work" — the strong-link thesis in six words
  • Nobel laureate Sydney Brenner: "peer review is hindering science. In fact, I think it has become a completely corrupt system"

Mental Models

  • Inversion — the weak-link framing is an inverted view of how scientific progress actually works
  • Second-Order Thinking — peer review's second-order effect was to legitimize a gatekeeping class without improving accuracy
  • Philosophy and Epistemology — raises fundamental questions about how scientific knowledge is validated and disseminated

Source

Adam Mastroianni, Experimental History (Substack) Original article

Backlinks