Vincent Müller and Nick Bostrom have just released a paper surveying the results of a poll of experts about future progress in artificial intelligence. The authors have also put up a companion site where visitors can take the poll and see the raw data. I just checked the site and so far only one individual has submitted a response. This provides an opportunity for testing the views of LW members against those of experts. So if you are willing to complete the questionnaire, please do so before reading the paper. (I have abstained from providing a link to the pdf to create a trivial inconvenience for those who cannot resist temptaion. Once you take the poll, you can easily find the paper by conducting a Google search with the keywords: bostrom muller future progress artificial intelligence.)
From the abstract of the paper:
Nearly one third of experts expect this development to be ‘bad’ or ‘extremely bad’ for humanity.
Where do they get this claim from? From the table in section 3.5 of the paper, it looks like they must have looked at the average probability that the experts gave for HLAI being bad or extremely bad (31%), but summarizing that as "nearly one third of experts expect ..." makes no sense. That phrasing suggests that there is a particular subset of the researchers surveyed, consisting of almost a third of them, that believes that the outcome would be bad or extremely bad. But you could get an average probability of 31% even if all the experts gave approximately the same probability distribution, and then there would be no way to pick out which third of them expect a bad result.
The most astonishing thing to me is what the paper gives as the responses to question 3, part B
"Assume for the purpose of this question that such HLMI will at some point exist. How likely do you then think it is that within (2 years / 30 years) thereafter there will be machine intelligence that greatly surpasses the performance of every human in most professions?"
EETN group, 30 years, median: 55%
What? They think that given a HLMI and 30 years, we have only a 55% chance make a SHLMI? Especially since they (on median) think it'll take 36 years to get 50% chance of HLMI in the first place (question 2). Do they really think that getting from human to superhuman is such a big step? The Top 100 had a somewhat lower rate (50% chance), but they at least thought it would take 56 years to get HLMI in the first place.
Maybe they're expecting that the only way the HLMI can be as effective as humans at most professions is by making up with massive efficiency where it does well for the messes it makes in parts it does poorly. But that doesn't fit with the broad generality of this machine. It can't have just a few things it does really well.
I think you're misreading the question. It's not about going from human to superhuman intelligence, it's about exceeding human performance in various professions once human-equivalent intelligence is achieved. A lot of professions involve more than just intelligence. Almost all professions involve a large amount of human interaction, as well as the ability to navigate spaces and situations that are designed for humans.
I am highly aware of the exact wording of the question, but thank you for checking. Your point makes it even more unbelieveable.
Things do depend on which professions you see it as handling adequately in the starting case, keeping in mind that it has to be adequate at 'most'. Also, what you consider a 'profession'. Lawyer, doctor, scientist, engineer, programmer... manager, carpenter, plumber, mason, police, fire fighter... receptionist, trucker, waiter, cook, retail clerk...
By the time you can be adequate at 'most' of those jobs, you've got the 'navigating human spaces' and 'interacting with humans' bit pretty well licked, so the difficulty of these matters is already handled by the time you start the clock.
If on the other hand you cut it off after the second ellipsis (the last few are jobs but not 'professions') then you might be able to be only halfway decent at navigating spaces heavily optimized for humans, but you have to already be positively ace at interacting with people.
If on the other hand you cut it off after the first ellipsis, and you're really just talking hard-core Professions with a capital P, then maaybe it can get away with being effectively immobile to begin with... but even in this extreme case, I can't see giving it the ability to walk, run, crawl, climb, shimmy, skoonch, and brachiate, or full equivalents, being a task that would span more than 30 years even taking the present as a starting point - and we haven't got HLMI today.
I'm going to actually link to the paper, because it was actually non-trivially difficult for me to find, and because this page is now the top result for your suggested search query.
You could compare with the existing poll results from http://lesswrong.com/lw/jj0/2013_survey_results/