The only thing that is required for an agent to provably have a utility function is that it has coherent preferences. The only thing that means is that the agent obeys the four axioms of VNM-rationality (which are really simple and would be really weird if they weren't satisfied). The Von Neumann - Morgenstern utility theorem states than B is preferred over A if and only if E(U(A)) < E(U(B)). And while it's true that when humans are presented with a choice between two lotteries, they sometimes pick the lottery that has a lower expected payout, this doesn't mean humans don't have a utility function, it just means that our utility function in those scenarios is not based entirely on the expected dollar value (it could instead be based on the expected payout where higher probabilities are given greater weight, for example).

I think utility functions can produce more behaviours than you give them credit for.

1. Humans don't have a utility function and make very incoherent decisions. Humans are also the most intelligent organisms on the planet. In fact, it seems to me that the less intelligent an organism is, the easier its behavior can be approximated with model that has a utility function!

The less intelligent organisms are certainly more predictable. But I think that the less intelligent ones actually can't be described by utility functions and are instead predictable for other reasons. A classic example is the Sphex wasp.

Some Sphex wasps drop a paralyzed insect near the opening of the nest. Before taking provisions into the nest, the Sphex first inspects the nest, leaving the prey outside. During the inspection, an experimenter can move the prey a few inches away from the opening. When the Sphex emerges from the nest ready to drag in the prey, it finds the prey missing. The Sphex quickly locates the moved prey, but now its behavioral "program" has been reset. After dragging the prey back to the opening of the nest, once again the Sphex is compelled to inspect the nest, so the prey is again dropped and left outside during another stereotypical inspection of the nest. This iteration can be repeated several times without the Sphex changing its sequence; by some accounts, endlessly.

So it looks like the wasp has a utility function "ensure the survival of its children" but in fact it's just following one of a number of fixed "programs". Whereas humans are actually capable of considering several plans and choosing the one they prefer, which I think is much closer to having a utility function. Of course humans are less predictable, but one would always expect intelligent organisms to be unpredictable. To predict an agent's actions you essentially have to mimic its thought processes, which will be longer for more intelligent organisms whether they use a utility function or not.

1. The randomness of human decisions seems essential to human success (on top of other essentials such as speech and cooking). Humans seem to have a knack for sacrificing precious lifetime for fool's errands that very occasionally create benefit for the entire species.

If trying actions at random produces useful results then a utility maximising AI will choose this course. Utility maximisers consider all plans and pick the one with the highest expected utility, and this can turn out to be one that doesn't look like it goes directly towards the goal. Eventually of course the AI will have to turn its attention towards its main goal. The question of when to do this is known as the exploration vs. exploitation tradeoff and there are mathematical results that utility maximisers tend to begin by exploring their options and then turn to exploiting their discoveries once they've learnt enough.

To define a utility function is to define a (direction towards a) goal. So a discussion of an AI with one, single, unchanging utility function is a discussion of an AI with one, single, unchanging goal. That isn't just unlike the intelligent organisms we know, it isn't even a failure mode of intelligent organisms we know. The nearest approximations we have are the least intelligent members of our species.

Again I think that this sort of behaviour (acting towards multiple goals) can be exhibited by utility maximizers. I'll give a simple example. Consider the agent who can by any 10 fruits from a market, and suppose its utility function is sqrt(number of oranges) + sqrt(number of apples). Then it buys 5 oranges and 5 apples (rather than just buying 10 apples or 10 oranges). The important thing about the example is the the derivative of the utility function is decreasing as the number of oranges increases, and so the more it has already the more it will prefer to buy apples instead. This creates a balance. This is just a simple example but by analogy it would be totally possible to create a utility function to describe a multitude of complex values all simultaneously.

1. Two agents with identical utility functions are arguably functionally identical to a single agent that exists in two instances. Two agents with utility functions that are not identical are at best irrelevant to each other and at worst implacable enemies.

Just like humans, two agents with different utility functions can cooperate through trade. The two agents calculate the outcome if they trade and the outcome if they don't trade, and they make the trade if the utility afterwards is higher for both of them. It's only if their utilities are diametrically opposed that they can't cooperate.

Humans don't have a utility function and make very incoherent decisions

Wait. Most of what I've read about utility functions applied to humans before anyone seriously talked about AGI. It doesn't have to be a simple function, and nobody may be able to algebraically express their utility function, but to the extent that any agent makes coherent goal-driven decisions, it has a utility function.

Humans that make incoherent decisions are either following a complex utility function that's hard to reverse-engineer, or have an inconstant utility function that changes across time.

I had the same thoughts after listening to the same talk. I think the advantage of utility functions, though, is that they are well-defined mathematical constructs we can reason about and showcase the corner cases that may pop up in other models but would also be easier to miss. AGI, just like all existing intelligences, may not be implemented with a utility function, but the utility function provides a powerful abstraction for reasoning about what we might call more loosely its "preference relation" that, by admitted contradictions, may risk us missing cases where the contradictions do not exist and the preference relation becomes a utility function.

The point being, for the purpose of alignment, studying utility functions makes more sense because your control method can't possibly work on a preference relation if it can't even work on the simpler utility function. That real preference relations contain things that prevent the challenges of aligning utility functions in existing intelligences instead provides evidence of how the problem might be solved (at least for some bounded cases).

The basic problem is the endemic confusion between the map, the UF as a way of modelling an entity, and the territory. the UF as an architectural feature that makes certain things happen.

It seems to you that entities with simple and obvious goal directed behaviour (as seen from the outside) have or need UFs, and entities that don't. don't. But there isn't a fixed connection between the way things seem from the outside, and the way they work.

From the outside, any system that succeeds in doing anything specialised can be thought of, or described as a relatively general purpose system that has been constrained down to a more narrow goal by some other system. For instance, a chess -playing system maybe described as a general purpose problem-solver that has been trained on chess. To say its UF defines a goal of winning at chess is the "map" view.

However, it might well be .. in terms of the territory ... in terms of what is going on inside the black box.. a special purpose system that has been specifically coded for chess, has no ability to do anything else, and therefore does not any kind of reward channel or training system to keep it focused on chess. So the mere fact that a system, considered from the outside as a black box, does some specific thing, is not proof that it has a UF, and therefore not a proof that anyone has succeeded in loading values or goals into its UF.

Taking an outside view of a system as possessing a UF (in the spirit or Dennett's "intentional stance") will only give correct predictions if everything works correctly. The essential point is that you need a fully accurate picture of what is going in inside a black box in order to predict its behaviour under all circumstances .. but pictures that are inaccurate in various ways can be good enough for restricted sets of circumstances.

Here's an analogy: suppose that machinery, including domestic appliances, were made of an infinitely malleable substance called Utlronium, say, and were constrained into some particular form, such a a kettle or toaster, by a further gadget called a Veeblefetzer. So long as a kettle functions as a kettle, I can regard it as Ultronium+Veeblefetzer ensemble. However, such ensembles support different counterfactuals to real kettles. For instance. if the veeblefetzer on my kettle fritzes it could suddenly reconfigure it into something else, a toaster or a spice rack -- but that is not possible for an ordinary kettle that is not made of Ultronium.

The converse case to an entity seeming to have a UF just because it fulfils some apparent purpose is an entity that seems not to have a UF because its behaviour is complex and perhaps seemingly random. A UF in the territory sense does not have to be simple, and a complex UF can include higher level goals, such as "seek variety" or "revise your lower level goals from time to time", so the lack of an obvious UF as judged externally does not imply the lack of a UF in the gold-standard sense of an actual component.

The actual possession of a UF is much more relevant to AI safety than being describable in terms of a UF. If an AI doesn't actually have a UF, you can't render it safe by fixing its UF.

I don't see any reason why AI has to act coherently. If it prefers A to B, B to C, and C to A, it might not care. You could program it to prefer that utility function.*

If not, maybe the A-liking aspects will reprogram B and C out of it's utility function, or maybe not. What happens would depend entirely on the details of how it was programmed.

Maybe it would spend all the universe's energy turning our future light cone from C to B, then from B to A, and also from A to C. Maybe it would do this all at once, if it was programmed to follow one "goal" before preceding to the next. Or maybe different parts of the universe would be in different stages, all at the same time. Think of it like a light-cone blender on pure.

Our default preferences seem about that coherent, but we're able to walk and talk, so clearly it's possible. It explains a lot of the madness and incoherence of the way the world is structured, certainly. Luckily, we seem to value coherence, or at least are willing to sacrifice on having our cake and eating it too when it becomes clear that we can't have it both ways. It's possible an subtly incoherent AGI would operate at cross purposes for a long time before discovering and correcting it's utility function, if it valued coherence.

However, MIRI isn't trying to program a sane AGI, not explore all possible ways an AI can be insane. Economists like to simplify human motives into idealized rational agents, because they are much, much simpler to reason about. The same is true for MIRI, I think.

I've given this sort of thing a little thought, and have a Evernote note I can turn into a LW post, if there is interest.

* I use the term "utility function broadly, here. I guess "programming" would be more correct, but even an A>B>C>A AI bears some rough resemblance to a utility function, even if it isn't coherent.

I think that there is hidden assumption that utility function is simple, so it could be easily calculated for any given position. So we have interaction of two algorithms: one extremely simple, utility function, and another is extremely complex (AGI). Most problems like paperclip maximiser results of such interaction.

The question which arise here, is it possible that utility function also will be very complex? For example as complex, as narrow AI? Could it help us in creating Friendly AI? Is known complexity of human values the same thing?

What would it mean for an AGI to not have a utility function, specifically what do you mean by "incoherently"?

The key issue for humans is not whether humans have utility functions, but whether it's useful to model humans as having utility functions.

To post title: Yes. See this discussion of quantum interference (decoherence) in human decision making: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0168045

Utility function +/- 25% against most uncertain prospect, in favor of prospect directly opposite most uncertain prospect. Add an additional +/- >5% as more information becomes available.

Somebody use that 25% in a back-prop algorithm already plz.

Just wondering if you've ever read an old economic article by Ron Heiner: The Origins of Predictable Behavior. 1984 (I think) Am. Econ. Rev. It's probably very sympathetic to your last paragraph. (and on a slightly different slant, the recent Qunata article about evolution: https://www.quantamagazine.org/20170314-time-dependent-rate-phenomenon-evolution-viruses/)

In fact, it seems to me that the less intelligent an organism is, the easier its behavior can be approximated with model that has a utility function!

Only because those organisms have fewer behaviors in general. If you put a human in an environment where its options and sensory inputs were as simple as those experienced by apes and cats, humans would probably look like equally simple utility maximizers.

I think you're getting stuck on the idea of one utility function. I like to think humans have many, many utility functions. Some we outgrow, some we "restart" from time to time. For the former, think of a baby learning to walk. There is a utility function, or something very much like it, that gets the baby from sitting to crawling to walking. Once the baby learns how to walk, though, the utility function is no longer useful; the goal has been met. Now this action moves from being modeled by a utility function to a known action that can be used as input to other utility functions.

As best as I can tell, human general intelligence comes from many small intelligences acting in a cohesive way. The brain is structured like this, as a bunch of different sections that do very specific things. Machine models are moving in this direction, with the Deepmind Go neural net playing a version of itself to get better a good example.