I think most people on this site (including me and you, private messaging/Dmytry) don't have any particular insight that gives them more information than those who seriously thought about this for a long time (like Eliezer, Ben Goertzel, Robin Hanson, Holden Karnofsky, Lukeprog, possibly Wei Dai, cousin_it, etc.), so our opinion on "who is right" is not worth much.

I'd much rather see an attempt to cleanly map out where knowledgeable people disagree, rather than polls of what ignorant people like me think.

Similarly, if two senior economists have a public disagreement about international trade and fiscal policy, a poll of a bunch of graduate students on those issues is not going to provide much new information to either economist.

(I don't really know how to phrase this argument cleanly, help and suggestions welcome, I'm just trying to retranscribe my general feeling of "I don't even know enough to answer, and I suspect neither to most people here")

I agree with HK that at this point SI should not be one of the priority charities supported by GiveWell, mainly due to the lack of demonstrated progress in the stated area of AI risk evaluation. If and when SI publishes peer-reviewed papers containing new insights into the subject matter, clearly demonstrating the dangers of AGI and providing a hard-to-dispute probability estimate of the UFAI takeover within a given time frame, as well as outlining constructive ways to mitigate this risk ("solve the friendliness problem" is too vague), GiveWell should reevaluate its stance.

On the other hand, the soon-to-be-spawned Applied Rationality org will have to be evaluated on its own merits, and is likely to have easier time of meeting GiveWell's requirements, mostly because the relevant metrics (of "raising the sanity waterline") can be made so much more concrete and near-term.

If you're actually interested in the answer to the question you describe yourself as wondering about, you might consider setting up a poll.

Conversely, if you're actually interested in expressing the belief that Holden is essentially correct while phrasing it as a rhetorical question for the usual reasons, then a poll isn't at all necessary.

I found HK's analysis largely sound (based on what I could follow, anyway), but it didn't have much of an effect on my donation practices. The following outlines my reasoning for doing what I do.

I have no feasible way to evaluate SIAI's work firsthand. I couldn't do that even if their findings were publicly available, and it's my default policy to reject the idea of donating to anyone whose claims I can't understand. If donating were a purely technical question, and if it came down to nothing but my estimate of SIAI's chances of actually making groundbreaking research, I wouldn't bet on them to be the first to build an AGI, never mind a FAI. (Also, on a more cynical note, if SIAI were simply an elaborate con job instead of a genuine research effort, I honestly wouldn't expect to see much of a difference.)

However, I can accept the core arguments for fast AI and uFAI to such a degree that I think the issue needs addressing, whatever that answer turns out to be. I view the AI risk PR work SIAI does as their most important contribution to date. Even if they never publish anything again, starting today, and even if they'll never have a line of code to show for anything, I estimate their net result to be positive simply for raising awareness about what looks to me like a legitimate concern. Someone should be asking those questions, and so far I haven't seen anyone else do that. To that end, I still estimate donating to SIAI to be worthwhile. At least for the time being.

I believe that SI is a valuable organisation and would be pleased if they were to keep their current level of funding.

I believe that withholding funds won't work very well and that they are rational and intelligent enough to sooner or later become aware of their shortcomings and update accordingly.

Having only read the headline, I came to this thread with the intention of saying that I agree with much of what he said, up to and potentially including withholding further funds from SI.

But then I read the post and find it's asking a different but related question, paraphrased as, "Why doesn't SI just lay down and die now that everyone knows none of their arguments have a basis in reality?" Which I'm inclined to disagree with.

I agree with Holden and additionally it looks like AGI discussions have most of the properties of mindkilling.

These discussions are about policy. They are about policy affecting medium-to-far future. These policies cannot be founded in reliably scientific evidence. Bayesian inquiry heavily depends on priors, and there is nowhere near anough data for tipping the point.

As someone who practices programming and has studied CS, I find Hanson and AI researchers and Holden more convincing than Eliezer_Yudkowsky or lukeprog. But this is more prior-based than evidence-based. Nearly all that the arguments by both sides do is just bringing a system to your priors. I cannot judge which side gives more odds-changing data because arguments from one side make way more sense and I cannot factor out the original prior dissonance with the other side.

The arguments about "optimization done better" don't tell us anything about position of fundamental limits to each kind of optimization; with a fixed computronium type it is not clear that any kind of head start would ensure that a single instance of AI would beat an instance based on 10x computronium older than 1 week (and partitioning the world's computer power for a month requires just a few ships with conveniently dropped anchors - we have seen it before, on a bit smaller scale). The limits can be further, but it is hard to be sure.

It may be that I fail to believe some parts of arguments because my priors are too strongly tipped. But Holden who has read most of the sequences without prior strong opinion wasn't convinced. This seems to support the theory of there being little mind-changing arguments.

Unfortunately, Transhumanist Wiki returns an error for a long time, so I cannot link to relatively recent "So you want to be a Seed AI Programmer" by Eliezer_Yudkowsky. If I say what I remembered best from there that made me more ready to discount SIAI-side priors it would be arguing with a fixed bottom line. I guess WebArchive version ( http://web.archive.org/web/20101227203946/http://www.acceleratingfuture.com/wiki/So_You_Want_To_Be_A_Seed_AI_Programmer ) should be quite OK - or is it missing important edits? Actually, it is a lot of content which puts Singularity arguments in slightly another light; maybe it should be either declared obsolete in public or saved at http://wiki.lesswrong.com/ for everyone who wants to read it.

I repeat once more that I consider most of the discussion to be caused by different priors and unshareable personal experiences. Personally me agreeing with Holden can give you only the information that a person like me can (not necessarily will) have such priors. If you agree with me, you cannot use me to check your reasons; if you disagree with me, I cannot convince you and you cannot convince me - not at our current state of knowledge.

Artificial Intelligence dates back to 1960. Fifty years later it has failed in such a humiliating way that it was not enough to move the goal posts; the old, heavy wooden goal posts have been burned and replaced with light weight portable aluminium goal posts, suitable for celebrating such achievements as from time to time occur.

Mainstream researchers have taken the history on board and now sit at their keyboards typing in code to hand-craft individual, focused solutions to each sub-challenge. Driving a car uses drive-a-car vision. Picking a nut and bolt from a component bin has nut-and-bolt vision. There is no generic see-vision. This kind of work cannot go FOOM for deep structural reasons. All the scary AI knowledge, the kind of knowledge that the pioneers of the 1960's dreamed of, stays in the brains of the human researchers. The humans write the code. Though they use meta-programming, it is always "well-founded" in the sense that level n writes level n-1, all the way down to level 0. There is no level n code rewriting level n. That is why it cannot go FOOM.

Importantly, this restraint is enforced by a different kind of self-interest than avoiding existential risk. The researchers have no idea how to write code with level n re-writing level n. Well, maybe they have the old ideas that never came close to working, but they know that if they venture into that toxic quagmire they will have nothing to show before their grant runs out, funders will think they wasted their grant on quixotic work, and their careers will be over.

Obviously past failure can lead to future success. Even a hundred and fifty years of failure can be trumped by eventual success. (Think of steam car work, which finally succeeded with the Stanley steamer, only to elbowed aside by internal combustion). So it is fair enough for the SI to say that past failure does not in itself rule out an AI-FOOM. But you cannot just ditch the history as though it never happened. We have learned a lot, most of it about how badly humans suck at programming computers. Current ideas of AI-risk are too thin to be taken seriously because there is no engagement with the history - researchers are working within a constraining paradigm because the history has dumped them in it, but the SI isn't worrying about how secure those constraints are, it is oblivious to them.

Suppose that SI now activates its AGI, unleashing it to reshape the world as it sees fit. What will be the outcome? I believe that the probability of an unfavorable outcome - by which I mean an outcome essentially equivalent to what a UFAI would bring about - exceeds 90% in such a scenario. I believe the goal of designing a "Friendly" utility function is likely to be beyond the abilities even of the best team of humans willing to design such a function. I do not have a tight argument for why I believe this.

My immediate reaction to this was "as opposed to doing what?" In this segment it seems like it is argued that SI's work, raising awareness that not all paths to AI are safe, and that we should strive to find safer paths towards AI, is actually making it more likely that an undesirable AI / Singularity will be spawned in the future. Can someone explain me how not discussing such issues and not working on them would be safer?

Just having that bottom line unresolved in Holden's post makes me reluctant to accept the rest of the argument.

I agree with Holden about everything.

Edit: Not that I'm complaining, but why is this upvoted? It's rather low on content.

SI is a very narrowly focused institute. If you don't buy the whole argument, there's very little reason to donate. I'm not sure SI should dissolve, I think they can reform. It's pretty obvious from their output that SI is essentially a machine ethics think tank. The obvious path to reform is greater pluralism and greater relevance to current debate. SI could focus on being the premiere machine ethics think tank, get involved in current ethical debates around the uses of AI, develop a more flexible ethical framework, and keep the Friendliness and Intelligence Explosion stuff as one possibility among many. This might allow them to grow and gain more resources (i.e., from the government, from military robotics companies wanting to appear responsible, etc), which would be a positive outcome for everyone. It'd also make it easier to donate, since instead of having to believe a narrow set of rather difficult to evaluate propositions, you'd simply have to value encouraging ethical debate around AI.

Please use more links. You should link to the post you're referring to, and probably a link to who Holden is and maybe even to what SI is.

I agree with HK that SIAI is not one of the best charities currently out there. I also agree with him that UFAI is a threat, and getting FAI is very difficult. I do not agree with HK's views on "tools" as opposed to "agents", primarily because I do not understand them fully. However I am fairly confident that if I did understand them I would disagree. I currently send all my charitable donations to AMF, but am open to starting to support SIAI when I see them publish more (peer-reviewed) material.

I believe SIAI believes it needs to present its arguments better because 1) many SIAI critics say things that indicate misunderstandings or miscommunication and 2) Lukeprog has not written up descriptions of all his relevant beliefs.

Broadly speaking, I agree with Holden, although possibly not his specific arguments. I'm not convinced that AI will appear in the manner SI postulates, and I have no real reason to believe that they will have an impact on existential probabilities. Similarly, I don't believe that donating to CND helped overt nuclear war.

Given that there are effective charities available which can make an immediate difference to people's lives, I would argue that concerned individuals should donate to those.