However, there's a problem - Eliezer is an expert on AI risk because he believes that AI risk is a bigger risk than nuclear war.

This isn't right. Eliezer got into the AI field because he wanted to make a Singularity happen sooner, and only later determined that AI risk is high. Even if Eliezer thought that nuclear war is a bigger risk than AI, he would still be in AI, because he would be thinking that creating a Singularity ASAP is the best way to prevent nuclear war.

Is there a sensible way for these people to correct for the fact that the people studying these risks are those that have high estimate of them in the first place?

I suggest that if you have the ability to evaluate the arguments on an object level, then do that, otherwise try to estimate P(E|H1) and P(E|H2) where E is the evidence you see and H1 is the "low risk" hypothesis (i.e., AI risk is actually low), H2 is the "high risk" hypothesis, and apply Bayes' rule.

Here's a simple argument for high AI risk. "AI is safe" implies that either superintelligence can't be created by humans, or any superintelligence we do create will somehow converge to a "correct" or "human-friendly" morality. Either of these may turn out to be true, but it's hard to see how anyone could (justifiably) have high confidence in either of them at this point in our state of knowledge.

As for P(E|H1) and P(E|H2), I think it's likely that even if AI risk is actually low, there would be someone in the world trying to make a living out of "crying wolf" about AI risk, so that alone (i.e., an apparent expert warning about AI risk) doesn't increase the posterior probability of H2 much. But what would be the likelihood of that person also creating a rationalist community and trying to "raise the sanity waterline"?

I think people should discount risk estimates fairly heavily when an organisation is based around doom mongering. For instance, The Singularity Institute, The Future of Humanity Institute and the Bulletin of the Atomic Scientists all seem pretty heavily oriented around doom. Such organisations initially attract those with high risk estimates, and they then actively try and "sell" their estimates to others.

Obtaining less biased estimates seems rather challenging. The end of the would would obviously be an unprecidented event.

The usual way of eliciting probability is with bets. However, with an apocalypse, this doesn't work too well. Attempts to use bets have some serious problems.

One way of testing this is to whether people are willing to discuss existential risk threats that cannot be solved by giving them money. Such comments do exist (see for example Stephen Hawking's comments about the danger of aliens). It is however interesting to note that he's made similar remarks about the threat of AI. (See e.g. here). I'm not sure whether such evaluations are relevant.

Also, I don't think it follows that people like Yudkowsky and Hellman necessarily decide to study the existential risks they do because they have a higher than average estimate for the threats in question. They may just have internalized the threats more. Most humans simply don't internalize the risks of existential risk in a way that alters their actions, even if they are willing to acknowledge high probabilities of problems.

I don't see a special problem...evaluate the arguments, try to correct for biases. Business as usual. Or do you suspect there is a new type of bias at work here?

People who work for http://nuclearrisk.org/ have a very high estimate of the probability that a nuclear war will destroy the world (by their estimates, if you are American and under 40, then nuclear war is the single most likely way in which you might die next year).

Nuclear weapons have been available on the "black market" (thanks to sloppy soviet handling practices) for decades, yet no terrorist or criminal group has ever used a nuclear fission initiation device. Nuclearrisk.org claims "terrorists may soon get their own button on the vest", citing Al-Qaeda's open desire to acquire nuclear weapons.

I am unable, if I assume fully honest and rational assessments, to rectify these points of fact with one another. They disagree with each other. Given the fact that, furthermore, many of these assessments of risk seem to carry the implicit assumption that if a single nuke is used, the whole world will start glowing in the dark ( see: http://news.stanford.edu/news/2009/july22/hellman-nuclear-analysis-071709.html for an example of this (Martin Hellman himself) ) -- well, it gets further absurd.

In other words; folks need to be careful, when crafting expert opinions, to avoid Déformation professionnelle.

Risk mitigation groups would gain some credibility by publishing concrete probability estimates of "the world will be destroyed by X before 2020" (and similar for other years). As many of the risks are a rather short event (think nuclear war / asteroid strike / singularity), the world will be destroyed by a single cause and the respective probabilities can be summed. I would not be surprised if the total probability comes out well above 1. Has anybody ever compiled a list of separate estimates?

On a related note, how much of the SIAI is financed on credit? Any group which estimates high risks of disastrous events should be willing to pay higher interest rates than market average. (As the expected amount of repayments is reduced by the nontrivial probability of everyone dying before maturity of the contract).

"because Eliezer is probably the world expert on AI risk"

There is no experts on the AI risk. There's nothing where to get expertise from. He read some SF, got caught on an idea, did not study (self study or otherwise) CS or any actual relevant body of knowledge to the point of producing anything useful, and he is a very convincing writer. The experts, you'll get experts in 2050. He's a dilettante.

People follow some sort of distribution on their risk estimates. Eliezer is just the far far off end of the bell curve on the risk estimate for AI, among those with writing skills. He does make some interesting points, but he's not a risk estimator.