Without knowing the content of your talk (or having time to Skype at present, apologies), allow me to offer a few quick points I would expect a reasonably well-informed, skeptical audience member to make (part-based on what I've encountered):

1) Intelligence explosion requires AI to get to a certain point of development before it can really take off (let's set aside that there's still a lot we need to figure out about where that point is, or whether there are multiple different versions of that point). People have been predicting that we can reach that stage of AI development "soon" since the Dartmouth conference. Why should we worry about this being on the horizon (rather than a thousand years away) now?

2) There's such a range of views on this topic by apparent experts in AI and computer science that an analyst might conclude "there is no credible expertise on "path/timeline to super intelligent AI". Why should we take MIRI/FHI's arguments seriously?

3) Why are mathematician/logician/philosophers/interdisciplinary researchers the community we should be taking most seriously when it comes to these concerns? Shouldn't we be talking to/hearing from the cutting edge AI "builders"?

4) (Related). MIRI (and also FHI, but not to such a 'primary' extend') focuses on developing theoretical safety designs, and friendly-AI/safety-relevant theorem proving and maths work ahead of any efforts to actually "build" AI. Would we not be better to be more grounded in the practical development of the technology - building, stopping, testing, trying, adapting as we see what works and what doesn't, rather than trying to lay down such far-reaching principles ahead of the technology development?

Holden Karnofsky post http://lesswrong.com/lw/cbs/thoughts_on_the_singularity_institute_si/ seems to be the best criticism of MIRI as an organisation.

Just to be clear: Are you looking for criticisms of the intelligence explosion hypothesis or of MIRI? The hypothesis is just one of many claims made by MIRI researchers, and not the most controversial one. Are you just going to be arguing for the plausibility of a (reasonably imminent) intelligence explosion, or are you planning on defending the whole suite of MIRIesque beliefs?

Some of the stuff I've posted - http://lesswrong.com/lw/ksa/the_metaphormyth_of_general_intelligence/ , http://lesswrong.com/lw/hvo/against_easy_superintelligence_the_unforeseen/ - could be used to build a good anti-MIRI steelman, but I've not seen them used.

The most convincing anti-MIRI argument? AI may not develop in the way you're imagining. The most convincing rebuttal? We only need a decent probability of that happening to justify worrying about it.

It may be more useful to ask actual critics what they think (rather than asking proponents what they think critics are trying to say). Robin Hanson criticizes foom here. I don't actually know what he thinks of MIRI.

Well, firstly its good that the crowd is savy, but it might still be wise to prepare for strawman/fleshman attacks as well as steelmanned ones.

These are some more plausible criticisms:

(1) Moore's law seems to be slowing - this could be a speedbump before the next paradigm takes over, or it could be the start of stagnation, in which case the singularity is postponed. Of course, if humanity survives the singularity will happen eventually anyway, but if it is hundreds of years in the future it would probably be wiser focussing on promoting rationality/genetic engineering/other methods of improving biological intelligence as well as cryonics in the short term, and leaving work on FAI to future generations.

(2) It could be argued that FAI and perhaps de novo AGI as well is simply so hard we will never get it done in time. Eventually neuromorphic AI/WBE/ brute force evolutionary simulations will be developed (assuming that exponential progress in these fields holds) and we would be better of preparing for this case, perhaps by developing empathic neuromorphic AI, or developing a framework for uploads and humans to live without a Malthusian race to the bottom.

(3) The budget/number of people involved of MIRI is tiny compared to google and other entities which could plausibly design AI. Therefore many would argue that MIRI cannot develop AI first, and instead should focus on outreach towards other, larger, groups.

(4) Gwern seems to be going further, and arguing that we should advocate that nations should suppress technology.

In all of these cases, some sort of rationality outreach would seem to be the alternative, so you could still spin that as a positive.

Well, I'm no Dymitry or XiXiDu, and not "especially capable" but why not give it a try.

Intelligence explosion pattern-matches pretty well to the religious ideas of Heaven/Hell, and cryonics to Limbo. Also note the dire warnings of the impending UFAI doom unless the FAI Savior is ushered by the righteous (called "rational") before it's too late. So one could dismiss the whole thing as a bad version of Christianity or some other religion.

I'd like there to be no real surprises.

It seems like surprises would be more valuable than just reciting info.

MIRI intends to make an AI that is provably friendly. This would require having a formal definition of friendliness that means exactly what it's supposed to mean, and then proving it. Either of those steps seems highly unlikely to be completed without error.

I can't think of rational arguments, even steelmanned ones, beyond those Holden already gave. Maybe I'm too close to the whole thing, but I think that when viewed rationally, MIRI is on pretty solid ground.

If I wanted to make people wary of supporting MIRI, I'd simply go ad hominem . Start with selected statements from supporters about how much MIRI is about Eliezer, and from Eliezer about how he can't get along with AI researchers, how he can't do straight work for more than two hours per day and how "this is a cult". Quote a few of the psychotic sounding parts from Eliezer's old autobiography piece. Paint him as a very skilled writer/persuader whose one great achievement was to get Peter Thiel to throw him a golden bone. Describe the whole Friendliness issue as an elaborate excuse from someone who claimed ability to code an AGI fifteen years ago, and hasn't.

Of course that's a lowly and unworthy style of argument, but it'd get attention from everyone there, and I wonder how you'd defend against it.

This comment is a poorly-organized brain dump which serves as a convenient gathering place for what I've learned after several days of arguing with every MIRI critic I could find. It will probably get it's own expanded post in the future, and if I have the time I may try to build a near-comprehensive list.

I've come to understand that criticisms of MIRI's version of the intelligence explosion hypothesis and the penumbra of ideas around it fall into two permeable categories:

Those that criticize MIRI as an organization or the whole FAI enterprise (people making these arguments may or may not be concerned about the actual IE) and those that attack object-level claims made by MIRI.

Broad Criticisms

1a) Why worry about this now, instead of in the distant future, given the abysmal performance of attempts to predict AI?

1b) Why take MIRI seriously when there are so many expert opinions that diverge?

1c) Aren't MIRI and LW just an Eliezer-worshipping cult?

1d) Is it even possible to do this kind of theoretical work so far in advance of actual testing and experimentation?

1e) The whole argument can be dismissed as it pattern matches other doomsday scenarios, almost all of which have been bullshit.

Specific Criticisms

2a) General intelligence is what we're worried about here, and it may prove much harder to build than we're anticipating.

2b) Tool AIs won't be as dangerous as agent AIs.

2c) Why not just build an Oracle?

2d) the FOOM will be distributed and slow, not fast and localized.

2e) Dumb Superintelligence, i.e. nothing worth of the name could possibly misinterpret a goal like 'make humans happy'

2f) Even FAI isn't a guarantee

2g) A self-improvement cascade will likely hit a wall at sub-superintelligent levels.

2h) Divergence Issue: all functioning AI systems have built-in sanity checks which take short-form goal statements and unpack them in ways that take account of constraints and context (???). It is actually impossible to build an AI which does not do this (???), and thus there can be no runaway SAI which is given a simple short-form goal and then carries it to ridiculous logical extremes (I WOULD BE PARTICULARLY INTERESTED IN SOMEONE ADDRESSING THIS).

A wrinkle in the foom argument, re: source code readability

There is a sense in which a programme can easily read its own source code. The whole point of a compiler is to scan and process source code. A C compiler can compile its own source code, providing it is written in C.

The wrinkle is the ability of a programme to knowingly read its own source code. Any running process can be put inside a sandbox or simulated environment, such that there is no surely technical way if circumventing it. A running process accesses its environment using system calls, for instance get_time() or open_file(), and it has to take their results on faith. The get_time() function doesn't have to return the real system time, and in a visualized process, attempts to access the file system do not access the real file system. There is no is_this_real() call, or at least, no unfakeable one. (Homoiconicity is no magic fix - even if a LISP programme can easily process it's own code once it has obtained it, it still has to trust the subroutine that advertises itself as returning it)

Therefore, a programme can easily be prevented from accessing and modifying its own code.

It could be argued that an intelligent agent could use social engineering to cajole a human operator into getting the source code out of a locked cabinet, or whatever. This is a variation on the standard MIRI claim that an AI could talk it's way out of a box. However,in this case the AI needs to talk it's way out before it starts to recursively self improve, because it needs it's source code. This suggests that an AI that is below a certain level of intelligence can be maintained there.