TB: Map and Territory (Yudkowsky)
Core Thesis
Rationality is the systematic practice of aligning your beliefs with reality and your actions with your goals — but it requires far more rigor than most people apply. The human brain is a flawed lens that can understand its own flaws, and that self-correcting capacity, not raw intelligence, is the source of good reasoning.
Key Takeaways
The flawed lens
- Our ability to control muscles is evolutionarily ancient; our ability to reason about our own reasoning is a recent, fragile innovation.
- The inner eye isn't sightless — it sees blurrily with systematic distortions. Recognizing the distortions is the first step to correcting them.
- You need a whole lot of rationality before it does anything but lead you into new and interesting mistakes.
Beliefs and evidence
- A belief is only worthwhile if you could, in principle, be persuaded to believe otherwise.
- The strength of a model is not what it can explain but what it can't. If you can explain all outcomes equally well, you have zero knowledge.
- That which can be destroyed by the truth should be; that which the truth nourishes should thrive.
Maps vs territory
- Mystery is a property of questions, not answers. Ignorance exists in the map, not the territory.
- Better to say "magic" than "emergence" or "complexity" — the latter create an illusion of understanding. Wiser to say "I don't know" and keep a placeholder.
- Applause lights: statements so trite that the reversal sounds abnormal. If the unreversed statement sounds profound, it may be conveying no information at all.
Rationality in practice
- Science is reflective reasoning about a more reliable process for making the content of your mind mirror the content of the world.
- Do not attempt to build long chains of deductive reasoning for complex plans — simply too fragile.
Mental Models
- The Map is Not the Territory — this book is an extended meditation on that concept; the title is literal
- Inversion — instead of defending your views, ask what would falsify them