LESSWRONG

Jun 13, 2013

106 comments

In general and across all instances I can think of so far, I do not agree with the part of your futurological forecast in which you reason, "After event W happens, everyone will see the truth of proposition X, leading them to endorse Y and agree with me about policy decision Z."

Example 1: "After a 2-year-old mouse is rejuvenated to allow 3 years of additional life, society will realize that human rejuvenation is possible, turn against deathism as the prospect of lifespan / healthspan extension starts to seem real, and demand a huge Manhattan Project to get it done." (EDIT: This has not happened, and the hypothetical is mouse healthspan extension, not anything cryonic. It's being cited because this is Aubrey de Grey's reasoning behind the Methuselah Mouse Prize.)

Alternative projection: Some media brouhaha. Lots of bioethicists acting concerned. Discussion dies off after a week. Nobody thinks about it afterward. The rest of society does not reason the same way Aubrey de Grey does.

Example 2: "As AI gets more sophisticated, everyone will realize that real AI is on the way and then they'll start taking Friendly AI development seriously."

Alternative projection: As AI gets more sophisticated, the rest of society can't see any difference between the latest breakthrough reported in a press release and that business earlier with Watson beating Ken Jennings or Deep Blue beating Kasparov; it seems like the same sort of press release to them. The same people who were talking about robot overlords earlier continue to talk about robot overlords. The same people who were talking about human irreproducibility continue to talk about human specialness. Concern is expressed over technological unemployment the same as today or Keynes in 1930, and this is used to fuel someone's previous ideological commitment to a basic income guarantee, inequality reduction, or whatever. The same tiny segment of unusually consequentialist people are concerned about Friendly AI as before. If anyone in the science community does start thinking that superintelligent AI is on the way, they exhibit the same distribution of performance as modern scientists who think it's on the way, e.g. Hugo de Garis, Ben Goertzel, etc.

Consider the situation in macroeconomics. When the Federal Reserve dropped interest rates to nearly zero and started printing money via quantitative easing, we had some people loudly predicting hyperinflation just because the monetary base had, you know, gone up by a factor of 10 or whatever it was. Which is kind of understandable. But still, a lot of mainstream economists (such as the Fed) thought we would not get hyperinflation, the implied spread on inflation-protected Treasuries and numerous other indicators showed that the free market thought we were due for below-trend inflation, and then in actual reality we got below-trend inflation. It's one thing to disagree with economists, another thing to disagree with implied market forecasts (why aren't you betting, if you really believe?) but you can still do it sometimes; but when conventional economics, market forecasts, and reality all agree on something, it's time to shut up and ask the economists how they knew. I had some credence in inflationary worries before that experience, but not afterward... So what about the rest of the world? In the heavily scientific community you live in, or if you read econblogs, you will find that a number of people actually have started to worry less about inflation and more about sub-trend nominal GDP growth. You will also find that right now these econblogs are having worry-fits about the Fed prematurely exiting QE and choking off the recovery because the elderly senior people with power have updated more slowly than the econblogs. And in larger society, if you look at what happens when Congresscritters question Bernanke, you will find that they are all terribly, terribly concerned about inflation. Still. The same as before. Some econblogs are very harsh on Bernanke because the Fed did not print enough money, but when I look at the kind of pressure Bernanke was getting from Congress, he starts to look to me like something of a hero just for following conventional macroeconomics as much as he did.

That issue is a hell of a lot more clear-cut than the medical science for human rejuvenation, which in turn is far more clear-cut ethically and policy-wise than issues in AI.

After event W happens, a few more relatively young scientists will see the truth of proposition X, and the larger society won't be able to tell a damn difference. This won't change the situation very much, there are probably already some scientists who endorse X, since X is probably pretty predictable even today if you're unbiased. The scientists who see the truth of X won't all rush to endorse Y, any more than current scientists who take X seriously all rush to endorse Y. As for people in power lining up behind your preferred policy option Z, forget it, they're old and set in their ways and Z is relatively novel without a large existing constituency favoring it. Expect W to be used as argument fodder to support conventional policy options that already have political force behind them, and for Z to not even be on the table.

37 points

Comments

sorted by

Highlighting new comments since Today at 3:06 AM

Rendering 50/106 comments show more

Halfwit

17 points

I do tend to think that Aubrey de Grey's argument holds some water. That is, it's not so much general society that will be influenced as wealthy elites. Elites seem more likely to update when they read about a 2x mouse. I suppose the Less Wrong response to this argument would be: how many of them are signed up for cryonics? But cryonics is a lot harder to believe than life extension. You need to buy pattern identity theory and nanotechnology and Hanson's value of life calculations. In the case of LE, all you have to believe is that the techniques that worked on the mouse will, likely, be useful in treating human senescence. And anyway, Aubrey hopes to first convince the gerontology community and then the public at large. This approach has worked for climate science and a similar approach may work for AI risk.

17 points

10 points

I suppose the Less Wrong response to this argument would be: how many of them are signed up for cryonics?

LessWrongers, and high-karma LessWrongers, on average seem to think cryonics won't work, with mean odds of 5:1 or more against cryonics (although the fact that they expect it to fail doesn't stop an inordinate proportion from trying it for the expected value).

On the other hand, if mice or human organs were cryopreserved and revived without brain damage or loss of viability, people would probably become a lot more (explicitly and emotionally) confident that there is no severe irreversible information loss. Much less impressive demonstrations have been enough to create huge demand to enlist in clinical trials before.

10 points

Paul Crowley

5 points

That number is the total probability of being revived taking into account x-risk among other things. It would be interesting to know how many people think it's likely to be technically feasable to revive future cryo patients.

5 points

16 points

For big jumpy events, look at the reactions to nuclear chain reactions, Sputnik, ENIGMA, penicillin, the Wright brothers, polio vaccine...

Then consider the process of gradual change with respect to the Internet, solar power, crop yields...

16 points

jkaufman

6 points

Some amount of bias (selection? availability?) there, in that part of why we think of your first paragraph examples is because they did make major news. There were probably others that were mostly ignored and so are much harder to think of. (Invention of the bifurcated needle, used for smallpox inoculations? What else has turned out to be really important in retrospect?)

6 points

Yvain

14 points

You mention Deep Blue beating Kasparov. This sounds look a good test case. I know that there were times when it was very controversial whether computers would ever be able to beat humans in chess - Wikipedia gives the example of a 1960s MIT professor who claimed that "no computer program could defeat even a 10-year-old child at chess". And it seems to me that by the time Deep Blue beat Kasparov, most people in the know agreed it would happen someday even if they didn't think Deep Blue itself would be the winner. A quick Google search doesn't pull up enough data to allow me to craft a full narrative of "people gradually became more and more willing to believe computers could beat grand masters with each incremental advance in chess technology", but it seems like the sort of thing that probably happened.

I think the economics example is a poor analogy, because it's a question about laws and not a question of gradual creeping recognition of a new technology. It also ignores one of the most important factors at play here - the recategorization of genres from "science fiction nerdery" to "something that will happen eventually" to "something that might happen in my lifetime and I should prepare for it."

14 points

rhollerith_dot_com

28 points

I know that there were times when it was very controversial whether computers would ever be able to beat humans in chess

Douglas Hofstadter being one on the wrong side: well, to be exact, he predicted (in his book GEB) that any computer that could play superhuman chess would necessarily have certain human qualities, e.g., if you ask it to play chess, it might reply, "I'm bored of chess; let's talk about poetry!" which IMHO is just as wrong as predicting that computers would never beat the best human players.

28 points

gwern

39 points

I thought you were exaggerating there, but I looked it up in my copy and he really did say that: pg684-686:

To conclude this Chapter, I would like to present ten "Questions and Speculations" about AI. I would not make so bold as to call them "Answers" - these are my personal opinions. They may well change in some ways, as I learn more and as AI develops more...

Question: Will there be chess programs that can beat anyone?

Speculation: No. There may be programs which can beat anyone at chess, but they will not be exclusively chess players. They will be programs of general intelligence, and they will be just as temperamental as people. "Do you want to play chess?" "No, I'm bored with chess. Let's talk about poetry." That may be the kind of dialogue you could have with a program that could beat everyone. That is because real intelligence inevitably depends on a total overview capacity - that is, a programmed ability to "jump out of the system", so to speak - at least roughly to the extent that we have that ability. Once that is present, you can't contain the program; it's gone beyond that certain critical point, and you just have to face the facts of what you've wrought.

I wonder if he did change his opinion on computer chess before Deep Blue and how long before? I found two relevant bits by him, but they don't really answer the question except they sound largely like excuse-making to my ears and like he was still fairly surprised it happened even as it was happening; from February 1996:

Several cognitive scientists said Deep Blue's victory in the opening game of the recent match told more about chess than about intelligence. "It was a watershed event, but it doesn't have to do with computers becoming intelligent," said Douglas Hofstadter, a professor of computer science at Indiana University and author of several books about human intelligence, including "Godel, Escher, Bach," which won a Pulitzer Prize in 1980, with its witty argument about the connecting threads of intellect in various fields of expression. "They're just overtaking humans in certain intellectual activities that we thought required intelligence. My God, I used to think chess required thought. Now, I realize it doesn't. It doesn't mean Kasparov isn't a deep thinker, just that you can bypass deep thinking in playing chess, the way you can fly without flapping your wings."...In "Godel, Escher, Bach" he held chess-playing to be a creative endeavor with the unrestrained threshold of excellence that pertains to arts like musical composition or literature. Now, he says, the computer gains of the last decade have persuaded him that chess is not as lofty an intellectual endeavor as music and writing; they require a soul. "I think chess is cerebral and intellectual," he said, "but it doesn't have deep emotional qualities to it, mortality, resignation, joy, all the things that music deals with. I'd put poetry and literature up there, too. If music or literature were created at an artistic level by a computer, I would feel this is a terrible thing."

And from January 2007:

Kelly said to me, "Doug, why did you not talk about the singularity and things like that in your book?" And I said, "Frankly, because it sort of disgusts me, but also because I just don't want to deal with science-fiction scenarios." I'm not talking about what's going to happen someday in the future; I'm not talking about decades or thousands of years in the future...And I don't have any real predictions as to when or if this is going to come about. I think there's some chance that some of what these people are saying is going to come about. When, I don't know. I wouldn't have predicted myself that the world chess champion would be defeated by a rather boring kind of chess program architecture, but it doesn't matter, it still did it. Nor would I have expected that a car would drive itself across the Nevada desert using laser rangefinders and television cameras and GPS and fancy computer programs. I wouldn't have guessed that that was going to happen when it happened. It's happening a little faster than I would have thought, and it does suggest that there may be some truth to the idea that Moore's Law [predicting a steady increase in computing power per unit cost] and all these other things are allowing us to develop things that have some things in common with our minds. I don't see anything yet that really resembles a human mind whatsoever. The car driving across the Nevada desert still strikes me as being closer to the thermostat or the toilet that regulates itself than to a human mind, and certainly the computer program that plays chess doesn't have any intelligence or anything like human thoughts.

39 points

Nominull

5 points

I suspect the thermostat is closer to the human mind than his conception of the human mind is.

5 points

14 points

Yes, people now believe that computers can beat people at chess.

14 points

6 points

I.e., they didn't update to expecting HAL immediately after, and they were right for solid reasons. But I think that the polls, and moreso polls of experts do respond to advancements in technology, e.g. on self-driving cars or solar power.

6 points

4 points

Do we have any evidence that they updated to expecting HAL in the long run? Normatively, I agree that ideal forecasters shouldn't be doing their updating on press releases, but people sometimes argue that press release W will cause people to update to X when they didn't realize X earlier.

4 points

AlanCrowe

12 points

I don't find either example convincing about the general point. Since I'm stupid I'll fail to spot that the mouse example uses fictional evidence and is best ignored

We are all pretty sick of seeing a headline "Cure for Alzheimer's disease!!!" and clicking through to the article only to find that it is cured in mice, knock-out mice, with a missing gene, and therefore suffering from a disease a little like human Alzheimer. The treatment turns out to be injecting them with the protein that the missing gene codes for. Relevance to human health: zero.

Mice are very short lived. We expect big boosts in life span by invoking mechanisms already present in humans and already working to provide humans with much longer life spans than mice. We don't expect big boosts in the life span of mice to herald very much for human health. Cats would be different. If pet cats started living 34 years instead of 17, their owners would certainly be saying "I want what Felix is getting."

The sophistication of AI is a tricky thing to measure. I think that we are safe from unfriendly AI for a few years yet, not so much because humans suck at programming computers, but because they suck in a particular way. Some humans can sit at a keyboard typing in hundreds of thousands of lines of code specific to a particular challenge and achieve great things. We can call that sophistication if we like, but it isn't going to go foom. The next big challenge requires a repeat of the heroic efforts, and generates another big pile of worn out keyboards. We suck at programming in the sense that we need to spend years typing in the code ourselves, we cannot write code that writes code.

Original visions of AI imagined a positronic brain in an anthropomorphic body. The robot could drive a car, play a violin, cook dinner, and beat you at chess. It was general purpose.

If one saw the distinction between special purpose and general purpose as the key issue, one might wonder: what would failure look like? I think the original vision would fail if one had separate robots, one for driving cars and flying airplanes, a second for playing musical instruments, a third to cook and clean, and fourth to play games such as chess, bridge, and baduk.

We have separate hand-crafted computer programs for chess and bridge and baduk. That is worse than failure.

Examples the other way.

After the Wright brothers people did believe in powered, heavier-than-air flight. Aircraft really took of after that. One crappy little hop in the most favourable weather and suddenly every-one's a believer.

Sputnik. Dreamers had been building rockets since the 1930's, and being laughed at. The German V2 was no laughing matter, but it was designed to crash into the ground and destroy things, which put an ugh field around thinking about what it meant. Then comes 1957. Beep, beep, beep! Suddenly every-one's a believer and twelve years later Buzz Aldrin and the other guy are standing on the moon :-)

The Battle of Cambrai is two examples of people "getting it". First people understood before the end of 1914 that the day of the horse-mounted cavalry charge was over. The Hussites had war wagons in 1420 so there was a long history of rejecting that kind of technology. But after event W1 (machine guns and barbed wire defeating horses) it only took three years before the first tank-mounted cavalry charge. I think we tend to miss understand this by measuring time in lives lost rather than in years. Yes, the adoption of armoured tanks was very slow if you count the delay in lives, but it couldn't have come much faster in months.

The second point is that first world war tanks were crap. The Cambrai salient was abandoned. The tanks were slow and always broke down, because they were too heavy and yet the armour was merely bullet proof. There only protection against artillery was that the gun laying techniques of the time were ill suited to moving targets. The deployment of tanks in the first world war fall short of being the critical event W. One expects the horrors of trench warfare to fade and military doctrine to go back to horses and charges in brightly coloured uniforms.

In reality the disappointing performance of the tanks didn't cause military thinkers to miss their significance. Governments did believe and developed doctrines of Blitzkreig and Cruiser tanks. Even a weak W can turn every-one into believers.

12 points

Locaha

12 points

http://www.fiercebiotech.com/special-reports/biopharmas-top-rd-spenders-2012

Example 1: "After a 2-year-old mouse is rejuvenated to allow 3 years of additional life, society will realize that human rejuvenation is possible, turn against deathism as the prospect of lifespan / healthspan extension starts to seem real, and demand a huge Manhattan Project to get it done."

A quick and dirty Google search reveals:

Cost of Manhattan Project in 2012 dollars: 30 billion

Pharma R&D budget in 2012: 70 billion

http://nuclearsecrecy.com/blog/2013/05/17/the-price-of-the-manhattan-project/

12 points

5 points

I think this is a good point, but I'd mention that real U.S. GDP is ~7 times higher now than then, aging as such isn't the focus of most pharma R&D (although if pharma companies thought they could actually make working drugs for it they would), and scientists' wages are higher now due to Baumol's cost disease.

5 points

Locaha

2 points

The point was, Pharma is spending enormous sums of money trying to fight individual diseases (and failing like 90% of the time), and people are proposing a Manhattan project to do something far more ambitious. But a Manhattan project is in the same order of magnitude as an individual drug. IOW, a Manhattan project won't be enough.

In any case, 3 years of additional life to a mouse won't be enough, because people can always claim that the intervention is not proportional to life span. What will do the trick is an immortalized mouse, as young at 15 years as it was at 0.5.

2 points

paulfchristiano

11 points

As far as I know, people have predicted every single big economic impact from technology well in advance, in the strong sense of making appropriate plans, making indicative utterances, etc. (I was claiming a few year's warning in the piece you are responding to, which is pretty minimal). Do you think there are counterexamples? You are claiming that a completely unprecedented will happen with very high probability. If you don't think that requires strong arguments to justify than I am confused, and if you think you've provided strong arguments I'm confused too.

I agree that AI has the potential to develop extremely quickly, in a way that only a handful of other technologies did. As far as I can tell the best reason to suspect that AI might be a surprise is that it is possible that only theoretical insights are needed, and we do have empirical evidence that sometimes people will be blindsided by a new mathematical proof. But again, as far as I know that has never resulted in a surprising economic impact, not even a modest one (and even in the domain of proofs, most of them don't blindside people, and there are strong arguments that AI is a harder problem than the problems that one person solves in isolation from the community---for example, literally thousands of times more effort has been put into it). A priori you might say "well, writing better conceptual algorithms is basically the same as proofs---and also sometimes blindsides people---and the total economic value of algorithms is massive at this point, so surely we would sometimes see huge jumps" but as far as I know you would be wrong.

There seems to be a big gap between the sort of problem on which progress is rapid and surprising, and the sort of problem on which progress would have an economic impact. There are a number of reasons to suspect this a priori (lots of people work on economically relevant problems, lots of people try to pay attention to development in those areas because it actually matters, economically relevant problems tend to have lots of moving pieces and require lots of work to get right, lots of people create working intermediate versions because those tend to also have economic impact, etc. etc.) and it seems to be an extremely strong empirical trend.

Like I said, I agree that AI has the potential to develop surprisingly quickly. I would say that 10% is a reasonable probability for such a surprising development (we have seen only a few cases of tech developments which could plausibly have rapid scale-up in economic significance; we also have empirical evidence from the nature of the relationship between theoretical progress and practical progress on software performance). This is a huge deal and something that people don't take nearly seriously enough. But your position on this question seems perplexing, and it doesn't seem surprising to me that most AI researchers dismiss it (and most other serious observers follow their lead, since your claim appears to be resting on a detailed view about the nature of AI, and it seems reasonable to believe people who have done serious work on AI when trying to evaluate such claims).

Making clear arguments for a more moderate and defensible conclusions seems like a good idea, and the sort of thing that would probably cause reasonable AI researchers to take the scenario more seriously.

11 points

5 points

As far as I know, people have predicted every single big economic impact from technology well in advance, in the strong sense of making appropriate plans, making indicative utterances, etc.

Is the thesis here that the surprisingness of atomic weapons does not count because there was still a 13-year delay from there until commercial nuclear power plants? It is not obvious to me that the key impact of AI is analogous to a commercial plant rather than an atomic weapon. I agree that broad economic impacts of somewhat-more-general tool-level AI may well be anticipated by some of the parties with a monetary stake in them, but this is not the same as anticipating a FOOM (X), endorsing the ideals of astronomical optimization (Y) and deploying the sort of policies we might consider wise for FOOM scenarios (Z).

5 points

paulfchristiano

4 points

Regarding atomic weapons:

Took many years and the prospect was widely understood amongst people who knew the field (I agree that massive wartime efforts to keep things secret are something of a special case, in terms of keeping knowledge from spreading from people who know what's up to other people).
Once you can make nuclear weapons you still have a continuous increase in destructive power; did it start from a level much higher than conventional bombing?

I do think this example is good for your case and unusually extreme, but if we are talking about a few years I think it still isn't surprising (except perhaps because of military secrecy).

but this is not the same as anticipating a FOOM (X), endorsing the ideals of astronomical optimization (Y) and deploying the sort of policies we might consider wise for FOOM scenarios (Z).

I don't think people will suspect a FOOM in particular, but I think they are open to the possibility to the extent that the arguments suggest it is plausible. I don't think you have argued against that much.

I don't think that people will become aggregative utilitarians when they think AI is imminent, but that seems like an odd suggestion at any rate. The policies we consider wise for a FOOM scenario are those that result in people basically remaining in control of the world rather than accidentally giving it up, which seems like a goal they basically share. Again, I agree that there is likely to be a gap between what I do and what others would do---e.g., I focus more on aggregate welfare, so am inclined to be more cautious. But that's a far cry from thinking that other people's plans don't matter, or even that my plans matter much more than everyone else's taken together.

4 points

Wei_Dai

4 points

I think I may be missing a relevant part of the previous discussion between you and Eliezer.

As far as I know, people have predicted every single big economic impact from technology well in advance, in the strong sense of making appropriate plans, making indicative utterances, etc.

By "people" do you mean at least one person, at least a few people, most people, most elites, or something else? What are we arguing about here, and what's the strategic relevance of the question?

(I was claiming a few year's warning in the piece you are responding to, which is pretty minimal).

Which piece?

There seems to be a big gap between the sort of problem on which progress is rapid and surprising, and the sort of problem on which progress would have an economic impact.

Would you consider Bitcoin to be a counterexample, at least potentially, if its economic impact keeps growing? (Although in general I think you're probably right, as it's hard to think of another similar example. There was some discussion about this here.)

4 points

paulfchristiano

3 points

By "people" do you mean at least one person, at least a few people, most people, most elites, or something else? What are we arguing about here, and what's the strategic relevance of the question?

I mean if you suggested "Technology X will have a huge economic impact in the near future" to a smart person who knew something about the area, they would think that was plausible and have reasonable estimates for the plausible magnitude of that impact.

The question is whether AI researchers and other elites who take them seriously will basically predict that human-level AI is coming, so that there will be good-faith attempts to mitigate impacts. I think this is very likely, and that improving society's capability to handle problems they recognize (e.g. to reason about them effectively) has a big impact on improving the probability that they will handle a transition to AI well. Eliezer tends to think this doesn't much matter, and that if lone heroes don't resolve the problems then there isn't much hope.

Which piece?

On my blog I made some remarks about AI, in particular saying that in the mainline people expect human-level AI before it happens. But I think the discussion makes sense without that.

Would you consider Bitcoin to be a counterexample, at least potentially, if its economic impact keeps growing? (Although in general I think you're probably right, as it's hard to think of another similar example. There was some discussion about this here.)

The economic impact of bitcoin to date is modest, and I expect it to increase continuously over a scale of years rather than jumping surprisingly.
I don't think people would have confidently predicted no digital currency prior to bitcoin, nor that they would predict that now. So if e.g. the emergence of digital currency was associated with big policy issues which warranted a pre-emptive response, and this was actually an important issue, I would expect people arguing for that policy response would get traction.
Bitcoin is probably still unusually extreme.

If Bitcoin precipitated a surprising shift in the economic organization of the world, then that would count.

I guess this part does depend a bit on context, since "surprising" depends on timescale. But Eliezer was referring to predictions of "a few years" of warning (which I think is on the very short end, and he thinks is on the very long end).

3 points

Qiaochu_Yuan

9 points

My version of Example 2 sounds more like "at some point, Watson might badly misdiagnose a human patient, or a bunch of self-driving cars might cause a terrible accident, or more inscrutable algorithms will do more inscrutable things, and this sort of thing might cause public opinion to turn against AI entirely in the same way that it turned against nuclear power."

9 points

diegocaleiro

4 points

At the Edge question 2009 6 people spoke of immortality (de Grey not included). and 17 people spoke of superintelligence/humans 2.0.

This seems like evidence for Aubrey's point of view.

Of all the things that 151 top scientists could think they'd live to see, that more than 10% converged on that without previous communication in that stuff is perplexing for anyone who was a transhumanist in 2005.

4 points

Douglas_Knight

5 points

without previous communication

No, these people all have long term relationships with Brockman/Edge, which even holds parties bringing them together.

5 points

Trevor_Blake

3 points

In general and across all instances I can think of so far, I do not agree with the part of your futurological forecast in which you reason, "After event W happens, everyone will see the truth of proposition X, leading them to endorse Y and agree with me about policy decision Z."

Sir Karl Popper came to the same conclusion in his 1963 book Conjectures and Refutations. So did Harold Walsby in his 1947 book The Domain of Ideologies. You're in good company.

3 points

Daniel_Burfoot

3 points

I agree that almost no actual individual will change his or her mind. But humanity as a whole can change its mind, as young impressionable scientists look around, take in the available evidence, and then commit themselves to a position and career trajectory based on that evidence.

3 points

jsteinhardt

4 points

Note that this would be a pretty slow change and is likely not fast enough if we only get, say, 5 years of prior warning.

4 points