Superintelligence: The Idea That Eats Smart People (2016)

Idle Words > Talks > Superintelligence This is the text version of a talk I gave on October 29, 2016, at Web Camp Zagreb [ video ] (45 mins) Superintelligence The Idea That Eats Smart People In 1945, as American physicists were preparing to test the atomic bomb, it occurred to someone to ask if such a test could set the atmosphere on fire. This was a legitimate concern. Nitrogen, which makes up most of the atmosphere, is not energetically stable. Smush two nitrogen atoms together hard enough and they will combine into an atom of magnesium, an alpha particle, and release a whole lot of energy: N 14 + N 14 ⇒ Mg 24 + α + 17.7 MeV The vital question was whether this reaction could be self-sustaining. The temperature inside the nuclear fireball would be hotter than any event in the Earth's history. Were we throwing a match into a bunch of dry leaves? Los Alamos physicists performed the analysis and decided there was a satisfactory margin of safety. Since we're all attending this conference today, we know they were right. They had confidence in their predictions because the laws governing nuclear reactions were straightforward and fairly well understood. Today we're building another world-changing technology, machine intelligence. We know that it will affect the world in profound ways, change how the economy works, and have knock-on effects we can't predict. But there's also the risk of a runaway reaction, where a machine intelligence reaches and exceeds human levels of intelligence in a very short span of time. At that point, social and economic problems would be the least of our worries. Any hyperintelligent machine (the argument goes) would have its own hypergoals, and would work to achieve them by manipulating humans, or simply using their bodies as a handy source of raw materials. Last year, the philosopher Nick Bostrom published Superintelligence , a book that synthesizes the alarmist view of AI and makes a case that such an intelligence explosion is both dangerous and inevitable given a set of modest assumptions. The computer that takes over the world is a staple scifi trope. But enough people take this scenario seriously that we have to take them seriously. Stephen Hawking , Elon Musk, and a whole raft of Silicon Valley investors and billionaires find this argument persuasive. Let me start by laying out the premises you need for Bostrom's argument to go through: The Premises Premise 1: Proof of Concept The first premise is the simple observation that thinking minds exist. We each carry on our shoulders a small box of thinking meat. I'm using mine to give this talk, you're using yours to listen. Sometimes, when the conditions are right, these minds are capable of rational thought. So we know that in principle, this is possible. Premise 2: No Quantum Shenanigans The second premise is that the brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components, just like we can simulate very basic neural anatomy today. Put another way, this is the premise that the mind arises out of ordinary physics. Some people like Roger Penrose would take issue with this argument, believing that there is extra stuff happening in the brain at a quantum level . If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept. Premise 3: Many Possible Minds The third premise is that the space of all possible minds is large. Our intelligence level, cognitive speed, set of biases and so on is not predetermined, but an artifact of our evolutionary history. In particular, there's no physical law that puts a cap on intelligence at the level of human beings. A good way to think of this is by looking what happens when the natural world tries to maximize for speed. If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster. But of course we know that there are all kinds of configurations of matter, like a motorcycle, that are faster than a cheetah and even look a little bit cooler. But there's no direct evolutionary pathway to the motorcycle. Evolution had to first make human beings, who then build all kinds of useful stuff. So analogously, there may be minds that are vastly smarter than our own, but which are just not accessible to evolution on Earth. It's possible that we could build them, or invent the machines that can invent the machines that can build them. There's likely to be some natural limit on intelligence, but there's no a priori reason to think that we're anywhere near it. Maybe the smartest a mind can be is twice as smart as people, maybe it's sixty thousand times as smart. That's an empirical question that we don't know how to answer. Premise 4: Plenty of Room at the Top The fourth premise is that there's still plenty of room for computers to get smaller and faster. If you watched the Apple event last night [where Apple introduced its 2016 laptops], you may be forgiven for thinking that Moore's Law is slowing down. But this premise just requires that you believe smaller and faster hardware to be possible in principle, down to several more orders of magnitude. We know from theory that the physical limits to computation are high . So we could keep doubling for decades more before we hit some kind of fundamental physical limit, rather than an economic or political limit to Moore's Law. Premise 5: Computer-Like Time Scales The penultimate premise is if we create an artificial intelligence, whether it's an emulated human brain or a de novo piece of software, it will operate at time scales that are characteristic of electronic hardware (microseconds) rather than human brains (hours). To get to the point where I could give this talk, I had to be born, grow up, go to school, attend university, live for a while, fly here and so on. It took years. Computers can work tens of thousands of times more quickly. In particular, you have to believe that an electronic mind could redesign itself (or the hardware it runs on) and then move over to the new configuration without having to re-learn everything on a human timescale, have long conversations with human tutors, go to college, try to find itself by taking painting classes, and so on. Premise 6: Recursive Self-Improvement The last premise is my favorite because it is the most unabashedly American premise. (This is Tony Robbins, a famous motivational speaker.) According to this premise, whatever goals an AI had (and they could be very weird, alien goals), it's going to want to improve itself. It's going to want to be a better AI. So it will find it useful to recursively redesign and improve its own systems to make itself smarter, and possibly live in a cooler enclosure. And by the time scale premise, this recursive self-improvement could happen very quickly. Conclusion: RAAAAAAR! If you accept all these premises, what you get is disaster!