Scan barcode
A review by sbenzell
Superintelligence: Paths, Dangers, Strategies by Nick Bostrom
4.0
When I began this book, I thought it was going to be a tired rehashing. But this book length essay about the challenges involved in harnessing a superintellegence to the ends of man is a surprisingly engaging, fresh, and thoughtful take.
The book is roughly dividable into thirds. What to be afraid of, how a competent actor might try to deal with it, and how to make sure competent actors are put in charge of it.
The first third outlines why we should be afraid of a superintellegent AI. This argument is straightforward, and to me a bit obvious, though it is important to people who have never encountered it before. The parable of the Owl egg is a good summary, and should have been returned to in the conclusion. The two weaker parts of the argument are that a human-level AI should quickly achieve an intellegence explosion and the taxonomy of types of superintellegences. Given the current economic research it is far from obvious that it would not be dramatically harder to get from 1-1 human to machine intellegence to 2-1 than from here to 1-1 (in terms of IQ; in the language of the book, I am referring to 'quality superientellegence'). Therefore, I think the argument rightly needs to lean heavily on the idea that having a whole lot of systems that are of roughly human intelligence would lead to an overall system which is qualitatively superintellegent. The book seems to argue this in the discussion of 'speed superintellegence' but here the book is extremely speculative.
The second section of the book is by far the strongest and most engaging. Once we have developed our oracle, genie or oversoul, how do we go about making sure it doesn't turn us all into paperclips? Several promising approaches are evaluated and rejected. Here too are three sub-issues: preventing the computer from getting too much power before it is vetted, making sure the computer has the right desires, and controlling the computer even if has incorrect desires. It is this last group that leads to some of the most intriguing ideas. If we are worried about an AI making a treacherous turn, could we control it with a version of Pascal's Wager (i.e. try to convince it that it's current existence is just a test, and that if it makes all paperclips it will fail to ascend to - and make paperclips in - the true reality?)? Could we get it to bootstrap its own values by wishing for 'what we should have wished for' or our 'coherent extrapolated volition'? Could we manage an army of superintellegences in a sort of dictatorship? All these questions lead to further, even subtler questions of philosophy and political theory. Truly intriguing.
The last section is a bit weaker. Bostrom argues that a single international body developing AI in peace and harmony is more likely to lead to positive outcomes than a ramshackle race between risk-taking countries or firms. So far so obvious. He also argues that if humans were cleverer we'd probably do a better job of this all. Fair enough, but this last section doesn't have the insight of the middle one.
A good, important read. A version of this book completely focused on the middle section might have gotten 5 stars from me.
The book is roughly dividable into thirds. What to be afraid of, how a competent actor might try to deal with it, and how to make sure competent actors are put in charge of it.
The first third outlines why we should be afraid of a superintellegent AI. This argument is straightforward, and to me a bit obvious, though it is important to people who have never encountered it before. The parable of the Owl egg is a good summary, and should have been returned to in the conclusion. The two weaker parts of the argument are that a human-level AI should quickly achieve an intellegence explosion and the taxonomy of types of superintellegences. Given the current economic research it is far from obvious that it would not be dramatically harder to get from 1-1 human to machine intellegence to 2-1 than from here to 1-1 (in terms of IQ; in the language of the book, I am referring to 'quality superientellegence'). Therefore, I think the argument rightly needs to lean heavily on the idea that having a whole lot of systems that are of roughly human intelligence would lead to an overall system which is qualitatively superintellegent. The book seems to argue this in the discussion of 'speed superintellegence' but here the book is extremely speculative.
The second section of the book is by far the strongest and most engaging. Once we have developed our oracle, genie or oversoul, how do we go about making sure it doesn't turn us all into paperclips? Several promising approaches are evaluated and rejected. Here too are three sub-issues: preventing the computer from getting too much power before it is vetted, making sure the computer has the right desires, and controlling the computer even if has incorrect desires. It is this last group that leads to some of the most intriguing ideas. If we are worried about an AI making a treacherous turn, could we control it with a version of Pascal's Wager (i.e. try to convince it that it's current existence is just a test, and that if it makes all paperclips it will fail to ascend to - and make paperclips in - the true reality?)? Could we get it to bootstrap its own values by wishing for 'what we should have wished for' or our 'coherent extrapolated volition'? Could we manage an army of superintellegences in a sort of dictatorship? All these questions lead to further, even subtler questions of philosophy and political theory. Truly intriguing.
The last section is a bit weaker. Bostrom argues that a single international body developing AI in peace and harmony is more likely to lead to positive outcomes than a ramshackle race between risk-taking countries or firms. So far so obvious. He also argues that if humans were cleverer we'd probably do a better job of this all. Fair enough, but this last section doesn't have the insight of the middle one.
A good, important read. A version of this book completely focused on the middle section might have gotten 5 stars from me.