Nexus

• Never summon powers you cannot control

-) The tendency to create powerful things with unintended consequesnces started not with the invention of the steam engine or AI but with the invention of religion

-) Power always stems from cooperation between large numbers of humans

-) The main argument of this book is that humankind gains enormous power by building large networks of cooperation, but the way these networks are built predisposes us to use that power unwisely. Our problem, then is a network problem.

-) It is an information problem. Information is the glue that holds networks together. But for 10 of thousands of year, Sapiens built and maintained large networks by inventing and spreading fictions, fantasies, and mass delusions.

-) The niave view of information: by gathering and processing much more information than individuals can, big networks achieve a better understanding of medicine, physics, economics, and numerous other fields, which makes the network not only powerful but wise. This view posits that in sufficient quantities information leads to truth, and truth in turn leads to both power and wisdom.

Ignorance– on this account– leads nowhere. While delusional or deceitful networks might arise in moments of crisis, in the long-term they are bout to lose to more clear-sighted and honest rivals.

The bigger the information network, the closer it must be to the truth.

-) Yet, with all the information currently circulating at breathtaking speeds, humanity is closer than ever to annihilating itself.

-) Given the magnitude of the danger, AI should be of interest to all human beings.

AI is the first technology in history that can make decisions and create new ideas by itself.

AI is not a tool, its an agent.

-) Homo Deus argued that the real hero of history has always been information, rather than Homo spaiens

-) What we usually think of as ideological and political conflicts often turn out to be clashes between opposing types of information networks.

-) Judged in terms of longevity, spread, and power, the Catholic Church has been perhaps the most successful institution in human history, despite – or perhaps because of – the relative weakness of its self-correcting mechanisms.

-) The rise of AI is arguably the biggest information revolution in history.

-) History isnt the study of the past, it is the study of change. History

JA: Communism as a negative example of AI alignment ?

-) A deep knowledge of history is vital to understand what is new about AI, how it is fundamentally different from printing presses and radio sets, and in what specific ways an AI dictatorship could be very unlike anything we have seen before.

-) The whole point of writing this book is that by making informed choices we can prevent the worst outcomes.

-) most information is not an attempt to represent reality. Most information does not represent anything

-) Truth is something that brings our attention to certain aspects of reality while inevitably ignoring other aspects.

-) Errors, lies, fantasies, and fictions are information too. Contrary to the Naive View of Information, information has no essential link to truth. Information creates new realities by tying together disparate things. Information doesnt necessarily inform us about things, it puts things in formation.

-) If the main job of information has been to represent reality accurately, it would have been hard to explain why the Bible became one of the most influential texts in history.

-) When we look at the history of information, we see a constant rise in connectivity without a concomitant rise in truthfulness or wisdom.

• We Sapiens rule the worl not because we are so wise but because we are the only animals that can cooperate flexibly in large numbers.
• About 70,000 years ago, Homo Sapiens bands began displaying an unprecedented capacity to cooperate with one another. Waht enabled different bands to cooperate is that evolutionary changes in brain structure and lunguistic abilities apparently gave Sapiens the aptitude to tell and believe fictional stories and to be deeply moved by them.
• Prior to the emergence of stories the universe contained just two levels of reality. Stories added a third. The first two levels are objective reality and subjective reality. The new level is inter-subjective reality.
• Luckily, since history is shaped by intersubjective stories, sometimes we can avert conflict and make peace by talking with people, changing the stories in which they and we believe, or coming up with a new story that everyone can accept.
• What the people at the top know, which nuclear physicists dont always, is that telling the truth about the universe is hardly the most efficient way to produce order among large numbers of people. When it comes to uniting people, fiction enjoys two inherent advantages over the truth. First, fiction can be made as simple as we like, whereas the truth tends to be complicated. Second, the truth is often painful and disturbing, and if we try to make it more comforting it will no longer be the truth.

• Long-term human memory is particularly adapted to retaining stories.
• Like mythology, bureaucracy too tends to sacrifice truth for order
• Mythology and bureaucracy are the twin pillary of every large-scale society.
• Artists usually work with a limited set of story lines that are rooted in our biology, but none of these dramas sheds much light on the workings of bureaucracy, because they have all been scripted by evolution millions of years before the emergence of documents and archives.
• All powerful information networks can do both good and ill, depending on how they are designed and used. Merely increasing the quantity of information in a network doesnt gaurantee its benevolence, or make it any easier to find the right balance between truth and order.
• Holy books like the Bible and the Quran are an information technology that is meant to both include all the vital information society needs and be free from all possibility of error. What happens when an information network believes itself to be utterly incapable of any error ? The history of allegedly infallible holy books highlights some of the limitations of all information networks and holds important lessons for the attemmpt to create infallible AIs in the 21st C.

• TruthGPT: In previous eras, such fantasies took a different form – religion. Historically, the most important function of religion as been to provide superhuman legitimacy for the social order. Studying the history of religion is thus highly relevant to present-day debates about AI.
• The book became an important religious technology in the 1st C BCE. After 10s of thousands of years in which gods spoke to humans via shamans, priests, prophets, oracles, and other human messengers, religious movements like Judaism began arguing that the gods speak through this novel technology of the book.
• Inevitably, the holy book spawned numerous interpretations which were far more consequential than the book itself. Writing down the word of God was supposed to limit the authority of the old priestly institution, but it gave rise to the authority of a new rabbinical institution. The attempt to bypass fallible human institutions by relying on a new information technology backfired, because of the need for a human institution to interpret the holy book.
• Judaism was shaped by the Talmud far more than by the Bible, and rabbinical arguments about the interpretation of the Talmud became even more important than the Talmud itself.
• The attempt to invest all authority in an infallible superhuman technology led the the rise of a new and extremely powerful human institution – the church.
• The Catholic Church interpreted Jesus's words in a way that allowed it to become the richest landowners in Europe, to launch violent crusades, and to establish murderous inquisitions.
• The European witch-hunt craze was a modern rather than a medieval phenomenon.
• The scientific revolution was launched by the discovery of ignorance. Its self-correcting mechanisms, not the technology of printing, was the engine of the scientific revolution.
• Self-correcting mechanisms are vital for the pursuit of truth, but are costly in terms of maintaining order.

• Demacracy and disctatorship are contrasting types of information networks.
• A common misconception is that in a demacracy everything is decided by majority vote. In fact, as little as possible is decided centrally, and only the relatively few decisions that must be made centrally should reflect the will of the majority. In a demacracy, if 99% of people want to dress in a particular way and worship a particular god, the remaining 1% should still be free to dress and worship differently.
• A demacracy is not a system in which a majority of any size can decide to exterminate unpopular minorities; it is a system in which there are clear limits on teh power of the center. There are two baskets of rights that are protected from the majorities grasp. One contains human rights. These rights enshrine the decentralized nature of demacracy, making sure that as loong as people dont harm anyone, they can live their lives as they see fit.
• Elections are NOT a method for discovering truth. They are a method for maintaining order by adjudicating between peoples conflicting desires.
• Subortinating society to a Ministry of Truth is likely to make things worse. The government is already the most powerful institution in developed societies, and it often has the greatest interst in distorting or hiding inconvenient facts. Allowing the governments to supervise the search for trurth is like appointing the fax to gaurd the chicken coop.
• Whereever the conversations that shape Ameerican politics now take place, its definitely not in Congress. Demacracies die not only people are not free to talk but also when people are not willing or able to listen.
• We dont know of any large-scale kingdom or empire that operated along democratic lines.
• Unlike premodern churches, which developed slowly over many centuries and therefore needed to be conservative and suspicious of rapid changes, modern totalitarian parties like the Nazi Party and the Soviet Communist Party were organized within a single generation around the promise to quickly revolutionize society.

• Democracy encourages information to flow through many indpendent channels rather than only through the center, and it allows many independent nodes to process the information and make decisions by themselves. Information freely circulates between private bussinesses, private media organizations, municipalities, sports associations, charitites, families, and individuals – withoug ever passing throught the office of a government minister.

  In contracts, totalitarianism wants ALL information to pass through the central hub and doesnt want any independent institutions making decisions on their own.

• Stalinism got close to world domination. It would be naive to assume that its disregard for truth doomed it to failure or that its ultimate collapse guarantees that such a system can never again arise. Information systems can reqch far with just a little truth and a lot of order.
• Now, humans will have to contend with digital mythmakers and digital bureaucrats. The main split in 21C politics might be not between democracy and totalitarianism but between human beings and nonhuman agents. A new Silicon Curtain may separate all humans from our unfathomable algorithmic overlords. People in all countries and walks of life – including even dictators – might find themselves subservient to an alien intelligence that can monitor everything we do while we have little idea what it is doing. The rest of this book, then, is dedicated to exploring whether such a Silicon Curtain is indeed descending on the world, and what liffe might look like when computers run our bureaucracies and algorithms invent new mythologies.

Computer As Source of Current Historical Revolution

• Computer revolution qualitatively more radical
• Computers ability to become more powerful than humans

jeremiads - a long mournful complaint, a list of woes

My goal with this book is to provide a more accurate historical perspective on the AI revolution.

AI Most Pressing Issue of our Time