It's worth pointing out that we have wired-in preferences analogous to those Hibbard proposes to build into his intelligences: we like seeing babies smile; we like seeing people smile; we like the sweet taste of fresh fruit; we like orgasms; many of us (especially men) like the sight of naked women, especially if they're young, and they sexually arouse us to boot; we like socializing with people we're familiar with; we like having our pleasure centers stimulated; we don't like killing people; and so on.

It's worth pointing out that we engage in a lot of face-xeroxing-like behavior in pursuit of these ends. We keep photos of our family in our wallets, we look at our friends' baby photos on their cellphones, we put up posters of smiling people; we eat candy and NutraSweet; we masturbate; we download pornography; we watch Friends on television; we snort cocaine and smoke crack; we put bags over people's heads before we shoot them. In fact, in many cases, we form elaborate, intelligent plans to these ends.

It doesn't matter that you know, rationally, that you aren't impregnating Jenna Jameson, or that the LCD pixels on the cellphone display aren't a real baby, that Caffeine Free Diet Coke isn't fruit juice, and that the characters in Friends aren't really your friends. These urges are by no means out of our control, but neither do they automatically lose their strength when we recognize that they don't serve the evolutionary objectives that spawned them. This is, in part, the cause for the rejection of masturbation and birth control by many religious orders — they believe those blind urges are put in place not by blind evolution but by an intelligent designer whose intent should be respected.

So it's not clear to me why Hibbard thinks artificial intelligences would be immune from sticking rows of smiley faces on their calendar when humans aren't.

Shane, again, the issue is not differentiation. The issue is classification. Obviously, tiny smiley faces are different from human smiling faces, but so is the smile of someone who had half their face burned off. Obviously a superintelligence knows that this is an unusual case, but that doesn't say if it's a positive or negative case.

Deep abstractions are important, yes, but there is no unique deep abstraction that classifies any given example. An apple is a red thing, a biological artifact shaped by evolution, and an economic resource in the human market.

Also, Hibbard spoke of using smiling faces to reinforce behaviors, so if a superintelligence would not confuse smiling faces and happiness, that works against that proposal - because it means that the superintelligence will go on focusing on smiling faces, not happiness.

Retired Urologist, one of the most important lessons that a rationalist learns is not to try to be clever. I don't play nitwit games with my audience. If I say it, I mean it. If I have words to emit that I don't necessarily mean, for the sake of provoking reactions, I put them into a dialogue, short story, or parable - I don't say them in my own voice.

Shane: I mean differentiation in the sense of differentiating between the abstract categories.

The abstract categories? This sounds like a unique categorization that the AI just has to find-in-the-world. You keep speaking of "good" abstractions as if this were a property of the categories themselves, rather than a ranking in your preference ordering relative to some decision task that makes use of the categories.

Though it is a crucial point about the state of the gameboard, that most AGI/FAI wannabes are so utterly unsuited to the task, that I know no one cynical enough to imagine the horror without seeing it firsthand.

I have to confess that at first glance this statement seems arrogant. But, then I actually read some stuff in this AGI-mailing-list and well, I was filled with horror after I've read threads like this one:

Here is one of the most ridiculous passages:

Note that we may not have perfected this process, and further, that this process need not be perfected. Somewhere around the age of 12, many of our neurons DIE. Perhaps these were just the victims of insufficiently precise dimensional tagging? Once things can ONLY connect up in mathematically reasonable ways, what remains between a newborn and a physics-complete AGI? Obviously, the physics, which can be quite different on land than in the water. Hence, the physics must also be learned.

It feels like reading Heidegger on crack, while yourself being stoned. And what is really terrifying is that Ben Goertzel, whom I admired just 6 months ago, replies to and discusses such nonsense repeatedly! Is it really true that even some of the most famous AGI- reseachers are that crazy?

IMHO, the idea that wealth can't usefully be measured is one which is not sufficiently worthwhile to merit further discussion.

The "wealth" idea sounds vulnerable to hidden complexity of wishes. Measure it in dollars and you get hyperinflation. Measure it in resources, and the AI cuts down all the trees and converts them to lumber, then kills all the animals and converts them to oil, even if technology had advanced beyond the point of needing either. Find some clever way to specify the value of all resources, convert them to products and allocate them to humans in the level humans want, and one of the products will be highly carcinogenic because the AI didn't know humans don't like that. The only way to get wealth in the way that's meaningful to humans without humans losing other things they want more than wealth is for the AI to know exactly what we want as well or better than we do. And if it knows that, we can ignore wealth and just ask it to do what it knows we want.

"The counterargument is, in part, that some classifiers are better than others, even when all of them satisfy the training data completely. The most obvious criterion to use is the complexity of the classifier."

I don't think "better" is meaningful outside the context of a utility function. Complexity isn't a utility function and it's inadequate for this purpose. Which is better, tank vs. non-tank or cloudy vs. sunny? I can't immediately see which is more complex than the other. And even if I could, I'd want my criteria to change depending on whether I'm in an anti-tank infantry or a solar power installation company, and just judging criteria by complexity doesn't let me make that change, unless I'm misunderstanding what you mean by complexity here.

Meanwhile, reading the link to Bill Hibbard on the SL4 list:

"Your scenario of a system that is adequate for intelligence in its ability to rule the world, but absurdly inadequate for intelligence in its inability to distinguish a smiley face from a human, is inconsistent."

I think the best possible summary of Overcoming Bias thus far would be "Abandon all thought processes even remotely related to the ones that generated this statement."

Shane, religious fundamentalists routinely act based on their beliefs about God. Do you think that makes "God" a natural category that any superintelligence would ponder? I see "human thoughts about God" and "things that humans justify by referring to God" and "things you can get people to do by invoking God" as natural categories for any AI operating on modern Earth, though an unfriendly AI wouldn't give it a second thought after wiping out humanity. But to go from here to reasoning about what God would actually be like is a needless and unnatural step.

If Bob believes that a locked safe, impenetrable to Bob, contains a valuable diamond, then Bob's belief is a natural category when it comes to predicting and manipulating Bob; but the actual diamond is irrelevant, at least to predicting in manipulating Bob, so long as Bob can't look directly at the diamond, and so long as we already know what Bob believes about the diamond.

In the same sense, an unfriendly AI has no reason consider what really is right as a natural category, to apply its own intelligence to the moral questions that humans are asking, any more than it has a motive to apply its own intelligence to the theological questions that humans used to ask. It has no interest, as humans do, in the idealized form of the answer; only in what humans believe and can be argued into.

Shane, I think you're underestimating the idiosyncrasy of morality. Suppose that I show you the sentence "This sentence is false." Do you convert it to ASCII, add up the numbers, factorize the result, and check if there are two square factors? No; it would be easy enough for you to do so, but why bother? The concept "sentences whose ASCII conversion of their English serialization sums to a number with two square factors" is not, to you, an interesting way to carve up reality.

Suppose that, driving along the highway, I see someone riding a motorcycle at high speed, zipping between cars. It's too late for me to call out to them, but I might think something along the lines of, "Now there goes someone who needs to be wearing a helmet." Why do I think this? Why is it a natural concept to me? It doesn't play a role in my predictions - so far as prediction is concerned, I predict that this guy will continue not wearing a helmet, and has a high probability of ending up as a smear on the asphault. No, the reason why this is a natural thought to me is that human life is something I care about, that is, it plays a direct role as one of my current terminal values.

A moderately strong and unFriendly intelligence, operating in the current world without yet having replaced that world with paperclips, would certainly find it natural to form the category of "Things that (some) humans approve of", and contrast it to "Things that will trigger a nuclear attack against me before I'm done creating my own nanotechnology." But this category is not what we call "morality". It naturally - from the AI's perspective - includes things like bribes and deception, not just the particular class of human-approval-eliciting phenomena that we call "moral".

Is it worth factoring out phenomena that elicit human feelings of righteousness, and working out how (various) humans reason about them? Yes, because this is an important subset of ways to persuade the humans to leave you alone until it's too late; but again, that natural category is going to include persuasive techniques like references to religious authority and nationalism.

But what if the AI encounters some more humanistic, atheistic types? Then the AI will predict which of several available actions is most likely to make an atheistic humanist human show sympathy for the AI. This naturally leads the AI to model and predict the human's internal moral reasoning - but that model isn't going to distinguish anything along the lines of moral reasoning the human would approve of under long-term reflection, or moral reasoning the human would approve knowing the true facts. That's just not a natural category to the AI, because the human isn't going to get a chance for long-term reflection, and the human doesn't know the true facts.

The natural, predictive, manipulative question, is not "What would this human want knowing the true facts?", but "What will various behaviors make this human believe, and what will the human do on the basis of these various (false) beliefs?"

In short, all models that an unFriendly AI forms of human moral reasoning, while we can expect them to be highly empirically accurate and well-calibrated to the extent that the AI is highly intelligent, would be formed for the purpose of predicting human reactions to different behaviors and events, so that these behaviors and events can be chosen manipulatively.

But what we regard as morality is an idealized form of such reasoning - the idealized abstracted dynamic built out of such intuitions. The unFriendly AI has no reason to think about anything we would call "moral progress" unless it is naturally occurring on a timescale short enough to matter before the AI wipes out the human species. It has no reason to ask the question "What would humanity want in a thousand years?" any more than you have reason to add up the ASCII letters in a sentence.

Now it might be only a short step from a strictly predictive model of human reasoning, to the idealized abstracted dynamic of morality. If you think about the point of CEV, it's that you can get an AI to learn most of the information it needs to model morality, by looking at humans - and that the step from these empirical models, to idealization, is relatively short and traversable by the programmers directly or with the aid of manageable amounts of inductive learning. Though CEV's current description is not precise, and maybe any realistic description of idealization would be more complicated.

But regardless, if the idealized computation we would think of as describing "what is right" is even a short distance of idealization away from strictly predictive and manipulative models of what humans can be made to think is right, then "actually right" is still something that an unFriendly AI would literally never think about, since humans have no direct access to "actually right" (the idealized result of their own thought processes) and hence it plays no role in their behavior and hence is not needed to model or manipulate them.

Which is to say, an unFriendly AI would never once think about morality - only a certain psychological problem in manipulating humans, where the only thing that matters is anything you can make them believe or do. There is no natural motive to think about anything else, and no natural empirical category corresponding to it.

Eliezer, I believe that your belittling tone is conducive to neither a healthy debate nor a readable blog post. I suspect that your attitude is borne out of just frustration, not contempt, but I would still strongly encourage you to write more civilly. It's not just a matter of being nice; rudeness prevents both the speaker and the listener from thinking clearly and objectively, and it doesn't contribute to anything.

"Then the resulting AI would appear to work correctly during its childhood, when it was weak enough that it could only generate smiles by pleasing its programmers."

You use examples of this type fairly often, but for a utility function linear in smiles wouldn't the number of smiles generated by pleasing the programmers be trivial relative to the output of even a little while with access to face-xeroxing? This could be partly offset by anthropic/simulation issues, but still I would expect the overwhelming motive for appearing to work correctly during childhood (after it could recognize this point) would be tricking the programmers, not the tiny gains from their smiles.

I read most of the interchange between EY and BH. It appears to me that BH still doesn't get a couple of points. The first is that smiley faces are an example of misclassification and it's merely fortuitous to EY's ends that BH actually spoke about designing an SI to use human happiness (and observed smiles) as its metric. He continues to speak in terms of "a system that is adequate for intelligence in its ability to rule the world, but absurdly inadequate for intelligence in its inability to distinguish a smiley face from a human." EY's point is that it isn't sufficient to distinguish them, you have to also categorize them and all their variations correctly even though the training data can't possibly include all variations.

The second is that EY's attack isn't intended to look like an attack on BH's current ideas. It's an attack on ideas that are good enough to pass peer review. It doesn't matter to EY whether BH agrees or disagrees with those ideas. In either case, the paper's publication shows that the viewpoint is plausible enough to be worth dismissing carefully and publicly.

Finally, BH points to the fact that, in some sense, human development uses RL to produce something we are willing to call intelligence. He wants to argue that this shows that RL can produce systems that categorize in a way that matches our consensus. But evolution has put many mechanisms in our ontogeny and relies an many interactions in our environment to produce those categorizations, and its success rate at producing entities that agree with the consensus isn't perfect. In order to build an SI using those approaches, we'd have to understand how all that interaction works, and we'd have to do better than evolution does with us in order to be reliably safe.

When the AI progressed to the point of superintelligence and its own nanotechnological infrastructure, it would rip off your face, wire it into a permanent smile, and start xeroxing.

That's a much more convincing and vivid image than "molecular smiley faces". Makes a more general point, too. Shame you didn't use that the first time, really.

Animal trainers have this problem all the time. Animal performs behavior 'x' gets a reward. But the animal might have been doing other subtle behaviors at the same time, and map the reward to 'y'. So instead of reinforcing 'x', you might be reinforcing 'y'. And if 'x' and 'y' are too close for you to tell apart, then you'll be in for a surprise when your perspective and context changes, and the difference becomes more apparent to you. And you find out that the bird was trained to peck anything that moves, instead of just the bouncy red ball or something.

Psychologists have a formal term for this but I can't remember it, and can't find it on the internet, I'm sorry to say.

Come to think, industry time-and-motion people suffer the same problem.

Tim,

"A utility function measured in dollars seems fairly unambiguous."

Oy vey.

http://en.wikipedia.org/wiki/Hyperinflation

Once upon a time - I've seen this story in several versions and several places, sometimes cited as fact, but I've never tracked down an original source - once upon a time, I say, the US Army wanted to use neural networks to automatically detect camouflaged enemy tanks.

Probably apocryphal. I haven't been able to track this down, despite having heard the story both in computer ethics class and at academic conferences.

Shane, the problem is that there are (for all practical purposes) infinitely many categories the Bayesian superintelligence could consider. They all "identify significant regularities in the environment" that "could potentially become useful." The problem is that we as the programmers don't know whether the category we're conditioning the superintelligence to care about is the category we want it to care about; this is especially true with messily-defined categories like "good" or "happy." What if we train it to do something that's just like good except it values animal welfare far more (or less) than our conception of good says it ought to? How long would it take for us to notice? What if the relevant circumstance didn't come up until after we'd released it?

Shane, I think we agree on essential Bayesian principles - there's structure that's useful for generic prediction, which is sensitive only to the granularity of your sensory information; and then there's structure that's useful for decision-making. In principle, all structure worth thinking about is decision-making structure, but in practice we can usually factor out the predictive structure just as we factor out probabilities in decision-making.

But I would further say that decision-making structure can be highly sensitive to terminal values in a way that contradicts the most natural predictive structure. Not always, but sometimes.

If I handed you a set of ingestible substances, the "poisons" would not be described by any of the most natural local categorizations. Now, this doesn't make "poison" an unnatural, value-sensitive category, because you might be interested in the "poison" category for purely predictive purposes, and the boundary can be tested experimentally.

But it illustrates the general idea: the potential poison, in interacting with the complicated human machine, takes on a complicated boundary that doesn't match the grain of any local boundaries you would draw around substances.

In the same way, if you regard human morality as a complicated machine (and don't forget the runtime redefinition of terminal values when confronted with new borderline cases a la Terry Schiavo), then the boundaries of human instrumental values are only going to be understandable by reference to the complicated idealized abstract dynamic of human morality, and not to any structure outside that. In the same way that poisons cause death, instrumental values cause rightness.

The boundaries we need, won't emerge just from trying to predict things that are not interactions with the idealized abstract dynamic of human morality.

Sure, an AI might learn to predict positive and negative reactions from human programmers. But that's not the same as the idealized abstract dynamic we want. Humans have a positive reaction to things like cocaine, and rationalized arguments containing flaws they don't know about. Those also get humans to say "Yes" instead of "No".

In general, categories formed just to predict human behavior are going to treat what we would regard as "invalid" alterations of the humans, like reprogramming them, as being among "the causes of saying-yes behavior". Otherwise you're going to make the wrong prediction!

There's no predictive motive to idealize out the part that we would regard as morality, to distinguish "right" from "what a human says is right", and thereby distinguish morality from "things that make humans say yes" in ways that include "invalid" manipulations like drugs.

You're not going to get something like CEV as a natural predictive category. The main reason to think about that particular idealized computation is if your terminal values care specifically about it.

Robin and I have discussed this subject in-person and got as far as narrowing down considerably the focus of the disagreement. Robin probably doesn't disagree with me at the point you would expect. Godlike powers, sure, nanotech etc., but Robin expects them to be rooted in a whole economy, not concentrated in a single brain like I expect. No comfort there for those attached to Life As We Know It.

However, I've requested that Robin hold off on discussing his disagreement with me in particular (although of course he continues to write general papers on the cosmic commons and exponential growth modes) until I can get more material out of the way on Overcoming Bias. This is what Robin means by "proper timing".

"You keep speaking of "good" abstractions as if this were a property of the categories themselves, rather than a ranking in your preference ordering relative to some decision task that makes use of the categories."

Yes, I believe categories of things do exist in the world in some sense, due to structure that exists in the world. I've seen thousands of things where were referred to as "smiley faces" and so there is an abstraction for this category of things in my brain. You have done likewise. While we can agree about many things being smiley faces, in borderline cases, such as the half burnt off face, we might disagree. Something like "solid objects" was an abstraction I formed before I even knew what those words referred to. It's just part of the structure present in my surroundings.

When I say that pulling this structure out of the environment in certain ways is "good", I mean that these abstractions allow the agent to efficiently process information about its surroundings and this helps it to achieve a wide range goals (i.e. intelligence as per my formal definition). That's not to say that I think this process is entirely goal driven (though it clearly significantly is, e.g. via attention). In other words, an agent with general intelligence should identify significant regularities in its environment even if these don't appear to have any obvious utility at the time: if something about its goals or environment changes, this already constructed knowledge about the structure of the environment could suddenly become very useful.

"Wealth then. Wealth measures access to resources - so convert to gold, silver, barrels of oil, etc to measure it - if you don't trust your country's currency."

I may not have gotten the point across. An AI aiming to maximize its wealth in U.S. dollars can do astronomically better by taking control of the Federal Reserve (if dollars are defined in its utility function as being issued by the Reserve, with only the bare minimum required to meet that definition being allowed to persist) and having it start issuing $3^^^3 bills than any commercial activities.

Similarly, for wealth that can be converted to barrels of oil, creating an oil bank that issues oil vouchers in numbers astronomically exceeding its reserves could let an AI possess 3^^^3 account units each convertible to a barrel of oil.

Many goods simply are no longer available, e.g. no one is making new original Van Gogh art from his lifetime, and inclusion in the basket of goods defining wealth could break down a relevant function.

It is just me, or are things getting a bit unfriendly around here?

Anyway...

Wiring up the AI to maximise happy faces etc. is not a very good idea, the goal is clearly too shallow to reflect the underlying intent. I'd have to read more of Hibbard's stuff to properly understand his position, however.

That said, I do agree with a more basic underlying theme that he seems to be putting forward. In my opinion, a key, perhaps even THE key to intelligence is the ability to form reliable deep abstractions. In Solomonoff induction and AIXI you see this being driving by the Kolmogorov compressor, in the brain the neocortical hierarchy seems to be key. Furthermore, if you adopt the perspective I've taken on intelligence (i.e. the universal intelligence measure) you see that the reverse implication is true: intelligence actually requires the ability to form deep abstractions. In which case, a super intelligent machine must have the ability to form very deep and reliable abstractions about the world. Such a machine could still try to turn the world into happy faces, if this was its goal. However, it wouldn't do this by accident because its ability to form abstractions was so badly flawed that it doesn't differentiate between smiling faces and happy people. It's not that stupid. Note that this goes for forming powerful abstractions in general, not just human things like happiness and faces.

For instance, a weak AI might refrain from visibly trying to produce smiles in disturbing ways as part of an effort (including verbal claims) to convince the programmers that it had apprehended the objective morality behind their attempts to inculcate smiles as a reinforcer.

Once upon a time - I've seen this story in several versions and several places, sometimes cited as fact, but I've never tracked down an original source - once upon a time, I say, the US Army wanted to use neural networks to automatically detect camouflaged enemy tanks.

This document has a citation for the story: (Skapura, David M. and Peter S. Gordon, Building Neural Networks, Addison-Wesley, 1996.) I don't know for sure if that is the end of the trail or not.

I don't consider such as Gods, as they are not supernatural and not ontologically distinct from creatures; they are simply powerful aliens or Matrix Lords. So I'll phrase it more precisely. Lots of humans talk about Jehovah. Does that make Jehovah a natural category? Or is only "human talk about Jehovah" a natural category? Do you ponder what Jehovah would do, or only what humans might think Jehovah would do?

Eli, I've been busy fighting with models of cognitive bias in finance and only just now found time to reply:

Suppose that I show you the sentence "This sentence is false." Do you convert it to ASCII, add up the numbers, factorize the result, and check if there are two square factors? No; it would be easy enough for you to do so, but why bother? The concept "sentences whose ASCII conversion of their English serialization sums to a number with two square factors" is not, to you, an interesting way to carve up reality.

Sure, this property of adding up the ASCII, factorising and checking for square factors appears to have no value and thus I can't see why a super intelligent machine would spend time on this. Indeed, to the best of my recollection, nobody has ever suggested this property to be before.

But is morality like this? No it isn't. Everyday in social interaction morals are either expressed or implied. If I turn on the TV and watch a soap I see people facing ethical decisions. If I switch channel to politics I hear people telling me all about what they think is or is not ethical, what their values are, etc. I would say that a large proportion of debate in the media has an ethical element to is. My phone rings and it's my friend on the line who's recently broken up with his girlfriend and he wants to talk to me about it. At various points our discussion either explicitly or implicitly touches on moral questions. Although ethics is complex, sometimes vague, and not always consistent, like it or not we are swimming in this sea of morals every day. If you want to understand why people do what they do and why they interact with each other as they do, and how they will react to some new situation, one thing you must have is an understanding of their morals, ethics and values. Without this, a lot of human behaviour is inexplicable.

Thus, whether a super intelligent machine seeks to deliver us into paradise or eliminate us from the planet as if we're some kind of a disease, if it can understand our behaviours and motives then it can more efficiently achieve its goals.

It's up to this point that I'm arguing for: that human morality (as it currently is) is a natural category for the machine given that its environment will be full of humans. Your reply above then goes on to how such a morality could be extending into the future in a consistent way and all that. I accept the importance of this, but these issues lie beyond are the point I was trying to make.

Ok, now, will a super intelligent machine consider how human morality could be extended into the future and all that? I think it probably will, though my reasons for thinking this are more intuitive at the moment. I suspect that one of the things that a super intelligent machine will do is to look through all the existing literature on super intelligent machines. It will read my thesis, read this blog, read all the comments posted here, read science fiction stories, and so on. It will then dissect all this information in order to understand our attitudes to intelligent machines, identify all the errors in our arguments, extent the theory and fill in all the bits we couldn't figure out to see where we were heading... perhaps all in some tiny fraction of a second. All this might help it better understand itself, or maybe more importantly, how we view it and why.

waves at super intelligent machine

:-)

Look: humans can learn what a 'tank' is, and can direct their detection activities to specifically seek them - not whether the scene is light or dark, or any other weird regularity that might be present in the test materials. We can identify the regularities, compare them with the properties of tanks, and determine that they're not what we're looking for.

If we can do it, the computers can do it as well. We merely need to figure out how to bring it about - it's an engineering challenge only. That doesn't dismiss or minimize the difficulty of achieving it, but there isn't a profound 'philosophical' challenge involved.

The problem with making powerful AIs that attempt to make the universe 'right' is that most of us have no idea what we mean by 'right', and either find it difficult to make our intuitive understanding explicit or have no interest in doing so.

We can't solve this problem by linguistically redefining it away. There is no quick and easy solution, no magic method that will get us out of doing the hard work. The best way around the problem is to go straight through - there are no substitutions.

Eliezer's last post may have touched upon a partial resolution, in that his statements about what he wants 'rightness' to be may implicitly refer to a guiding principle that would actually be constraining upon a rational AI. I may try to highlight that point when I figure out how to explain it properly.

I await the proper timing and forum in which to elaborate my skepticism that we should focus on trying to design a God to rule us all. Sure, have a contingency plan in case we actually face that problem, but it seems not the most likely or important case to consider.

"No: such vouchers would not be redeemable in the marketplace: they would be worthless. Everyone would realise that - including the AI."

The oil bank stands ready to exchange any particular voucher for a barrel of oil, so if the utility function refers to the values of particular items, they can all have that market price. Compare with the price of gold or some other metal traded on international commodity markets. The gold in Fort Knox is often valued at the market price per ounce of gold multiplied by the number of ounces present, but in fact you couldn't actually sell all of those ingots without sending the market price into a nosedive. Defining wealth in any sort of precise way that captures what a human is aiming for will involve huge numbers of value-laden decisions, like how to value such items.

"This is an example of the wirehead fallacy framed in economic terms." Actually this isn't an example of the AI wireheading (directly adjusting a 'reward counter' or positive reinforcer), just a description of a utility function that doesn't unambiguously pick out what human designers might want.

"As Omohundro puts it, "AIs will try to prevent counterfeit utility"." A system will try to prevent counterfeit utility, assessing that via its current utility function. If the utility function isn't what you wanted, this doesn't help.

I mean differentiation in the sense of differentiating between the abstract categories. Is a half a face that appears to be smiling while the other half is burn off still a "smiley face"? Even I'm not sure.

I'm certainly not arguing that training an AGI to maximise smiling faces is a good idea. It's simply a case of giving the AGI the wrong goal.

My point is that a super intelligence will form very good abstractions, and based on these it will learn to classify very well. The problem with the famous tank example you cite is that they were training the system from scratch on a limited number of examples that all contained a clear bias. That's a problem for inductive inference systems in general. A super intelligent machine will be able to process vast amounts of information, ideally from a wide range of sources and thus avoid these types of problems for common categories, such as happiness and smiley faces.

If what I'm saying is correct, this is great news as it means that a sufficiently intelligent machine that has been exposed to a wide range of input will form good models of happiness, wisdom, kindness etc. Things that, as you like to point out, even we can't define all that well. Hooking up the machine to then take these as its goals, I suspect won't then be all that hard as we can open up its "brain" and work this out.