To put my point a bit differently I think your argument would disappear if you tried to define imagine/reason more precisely.
Humans certainly aren't perfect at imagining. In fact if you ask most people to imagine a heavy object and a much heavier object falling they will predict the much heavier object hits first and I can give a host of other examples of the same thing. So certainly we can't require perfection to count as imagining or reasoning.
Neither do we want to define reasoning in terms of speaking english sentences or anything similar. I mean we can imagine the possibility of intelligent alien blobs of gas that have no language at all (or at least that we can understand). Besides using this definition would just make the claims trivial.
The best approach I can think of to making these terms more precisce is by defining reasoning ability in terms of ability to respond in a survival enhancing way to a diverse set of conditions. However, when we try to define it this way it is far from clear that complexes of genes aren't just as good at reasoning as people are.
Sure it's weird that something like this might be the case but it's damn weird that induction works at all so why not believe it can be captured in a large number of relatively simple heuristics.
Actually there is an (almost explicit) contradiction in the way most religions talk about morality and god.
I'm most familiar with christianity (specifically catholicism) but I believe the same goes for most major monotheistic religions.
1) They claim that morality arises from god, i.e., they wish to define morality as obeying god's commands.
2) "God is good," is an explicit part of their doctrines.
The tension here is obvious. Clearly the members of the religion take themselves to be saying something substantive and meaningful when they all intone "God is good" but yet if they really believed that morality was merely obedience to god then you aren't actually asserting anything of substance when you claim that god is good.
So not only does the theory that morality come from god have some philosophical problems the people who advocate it don't even really seem to believe it.
I just want to remark that it is far from obvious on apriori grounds that there is no elegant general AI algorithm that will solve all the other problems quite nicely. We've only learned this by the continued failure to find such an algorithm or anything like it by the AI community and the continued small successes of more specific less elegant approaches.
But in this era of neurology, one ought to be aware that thoughts are existent in the universe; they are identical to the operation of brains.
Really? I'm aware that physical outputs are totally determined by physical inputs. Neurology can tell us what sorts of physical causes give rise to what sorts of physical effects. We even have reason to believe that thoughts can be infered from the physical state of the brain in a lawlike fashion but this surely doesn't let us infer that thoughts are IDENTICAL to the operation of brains. Merely that they always go together in the actual world.
In particular (as chalmers argues most convincingly) there is nothing contradictory about imagining we have the same physical state but totally different experiences, i.e., that it might feel like something totally different to be us. Worse we actually only know about the physical world through our experiences so you can't simplify the problem purely down to a physical one. It's actually the more complicated one of why the lawlike relationship between experiences and physical facts is set up right for us to have knowledge. It could have been that the physical world was all the same but we just had totally incoherent experiences.
I think the issue is that science tells us that there is a certain kind of explanation, not just that there is some explanation.
Most people (at some level) want to believe in something bigger than themselves that cares about human concerns. The reason that magic is exciting so long as it doesn't have a scientific explanation is that it holds out the possibility that is responds to human concerns.
Think for a moment about the differences between a science fiction book and fantasy book. Both of them endow their major characthers to accomplish astounding feats through essentially unexplained means but the the hyperdrive button or the transporter has a very different feeling to it than the spell. Why? Because the scifi gadgets presumably work on micro-physical laws that are indifferent to people's emotions while spells are depicted as responding to human emotions like hate, love and need. People want to believe in a universe that cares about these human level concerns and that's why they believe in ghosts, god, psychics etc...
Now I think that if you said 'science' that would dispel the interest from most folk but wouldn't do much to dispel the interest of scientists. I mean why should common folk be curious about this since saying 'science' means it's going to be just another of the innumerable technologies they don't really understand but see every day. By making the phenomena seem like magic you held out the hope for a second of something that would be much more interesting to them (they are rationally sure they won't understand any scientific explanation anyway).
In short I think this is just a complex way of illustrating the fact that many people would like to believe in something other than materialism.
To quibble just a bit I think that it is occasionally (tho probably not in the terminator examples the paper briefly mentioned) reasonable to use a very throughly fleshed out fictional account as evidence of plausibility. I mean by giving a detailed narrative you rule out the possibility the idea is internally incoherent or requires some really really implausible things to be true.
Still, I don't think this is a very strong effect and is overestimated all the time by people who think that literature gives more than entertainment/enjoyment but actually gives insight.
You need to do a lot more to demonstrate irrationality than this. Obviously, as other commenters have pointed out, there are an infinite number of rules that agree with any given finite sequence of experimental results so obviously you can never conclusively demonstrate that your rule is indeed the correct one. Moreover, you can't even be 'bias free' in the sense of assigning all possible rules the same probability unless you want to assign each rule probability 0.
Now you might be tempted to just give up at this point but this is exactly the same problem we face when doing science. We have an infinite number of possible rules that extend the results we have seen so far and we need to guess which is most likely. Amazingly we do it pretty well but justifying it seems impossible, it's the classical philosophical problem of induction.
In short it's not clear anyone is 'wrong'. Maybe they have a good initial probability distribution for what sorts of rules people normally pick. Heck it's not even clear what it means to be 'wrong' in this sense, i.e., having an implausible a priori probability distribution
Speaking as a mathematician (well a grad student) I can positively say I frequently see 'by definition' used in arguments and use it myself in a substantive valid fashion. Sure, definitions don't support inductive inference but that doesn't mean they are always trivial. Fermat's last theorem follow by definition from the definition of the integers but it's certainly not a trivial fact that it does so. While rarely quite so complex arguments about philosophy, politics and other things can sometimes benefit from the nonobvious manipulation of definitions.
Also the notion of something following 'by definition' is of incredible use of rebutting a great deal of misguided philosophy. Quite frequently in philosophy one will see an analysis of a concept like, life, knowledge, or morality claiming to be a explication of the term we use in everyday language. However, it can be very useful to point out that no matter what the theoretical virtues of reducing 'moral good' to 'that which produces moral emotions in us' it's simply definitionally false. To the extent that we have a coherent notion of 'moral good' the sense that it refers to interpersonal facts that are more than mere feelings is inseperable from the concept. The word might change meaning but a theory which says there is nothing more to morality than moral feelings is by definition claiming that there are no moral facts not offering a materialist account of them.
Ironically in making this point I'm somewhat agreeing with Eliezer. The real benefit in using the definition like I did above was to combat the abuse of that definition in the original argument. Definitions are dangerous in arguments but not because people are inclined to say 'by definition' when they mean 'damn right' but because they let the presenter shift the flaw in their reasoning far away from the controversial results. Unless your used to evaluating complex mathematical arguments (and even then) a subtle flaw in a definition at the begining of an argument that has been forgotten by the time the conclusion is reached can be extremely misleading. Worse it harnesses the social awkwardness of being pedantic and insisting on details and rigor before the person has been allowed to get into the meat of their ideas to squash your ability to find the flaws (obviously these are side effects not purposeful choices)