TheRiseAndFallOfPeerReview
Great!
https://www.experimental-history.com/p/the-rise-and-fall-of-peer-review
- Peer review: the most massive experiment ever run, and it included every scientist on Earth. It was run poorly
- Peer review started after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals.
- The results are in. It failed.
- Peer review was a huge, expensive intervention. Huge interventions should have huge effects. if peer review improved science, that should be pretty obvious, and we should be pretty upset and embarrassed if it didn’t.
- All we can say from these big trends is that we have no idea whether peer review helped, it might have hurt, it cost a ton, and the current state of the scientific literature is pretty abysmal.
- It doesnt work: Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this. In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time
- Once a paper gets published, we shred the reviews. A few journals publish reviews; most don't. Nobody cares to find out what the reviewers said or how the authors edited their paper in response, which suggests that nobody thinks the reviews actually mattered in the first place.
- scientists take unreviewed work seriously without thinking twice.
- I think we had the wrong model of how science works. We treated science like it’s a weak-link problem where progress depends on the quality of our worst work. If you believe in weak-link science, you think it’s very important to stamp out untrue ideas—ideally, prevent them from being published in the first place. You don’t mind if you whack a few good ideas in the process, because it’s so important to bury the bad stuff. But science is a strong-link problem: progress depends on the quality of our best work.