Start the AI in a sandbox universe, like the "game of life". Give it a prior saying that universe is the only one that exists (no universal priors plz), and a utility function that tells it to spell out the answer to some formally specified question in some predefined spot within the universe. Run for many cycles, stop, inspect the answer.

Start the AI in a sandbox universe. Define its utility function over 32-bit integers. Somewhere inside the sandbox, put something that sets its utility to INT_MAX utility, then halts the simulation. Outside the sandbox, leave documentation of this readily accessible. The AI should never try to do something elaborately horrible, because it can get max utility easily enough from inside the simulation; if it does escape the box, it should go back in to collect its INT_MAX utility.

Oracle AI - its only desire is to provide the correct answer to yes or no questions posed to it in some formal language (sort of an ueber Watson).

So, here's my pet theory for AI that I'd love to put out of it's misery: "Don't do anything your designer wouldn't approve of". It's loosely based on the "Gandi wouldn't take a pill that would turn him into a murderer" principle.

A possible implementation: Make an emulation of the designer and use it as an isolated component of the AI. Any plan of action has to be submitted for approval to this component before being implemented. This is nicely recursive and rejects plans such as "make a plan of action deceptively complex such that my designer will mistakenly approve it" and "modify my designer so that they approve what I want them to approve".

There could be an argument about how the designer's emulation would feel in this situation, but.. torture vs. dust specks! Also, is this a corrupted version of ?

The Philosophical Insight Generator - Using a model of a volunteer's mind, generate short (<200 characters, say) strings that the model rates as highly insightful after read each string by itself, and print out the top 100000 such strings (after applying some semantic distance criteria or using the model to filter out duplicate insights) after running for a certain number of ticks.

Have the volunteer read these insights along with the rest of the FAI team in random order, discuss, update the model, then repeat as needed.

The AI gets positive utility from having been created, and that is the whole of its utility function. It's given a sandbox full of decision-theoretic problems to play with, and is put in a box (i.e. it can't meaningfully influence the outside world until it has superhuman intelligence). Design it in such a way that it's initially biased toward action rather than inaction if it anticipates equal utility from both.

Unless the AI develops some sort of non-causal decision theory, it has no reason to do anything. If it develops TDT, it will try to act in accordance with what it judges to be the wishes of its creators, following You're In Newcomb's Box logic--it will try to be the sort of thing its creators wished to create.

Define "Interim Friendliness" as a set of constraints on the AI's behavior which is only meant to last until it figures out true Friendliness, and a "Proxy Judge" as a computational process used to judge the adequacy of a proposed definition of true Friendliness.

Then there's a large class of Friendliness-finding strategies where the AI is instructed as follows: With your actions constrained by Interim Friendliness, find a definition of true Friendliness which meets the approval of the Proxy Judge with very high probability, and adopt that as your long-term goal / value system / decision theory.

In effect, Interim Friendliness is there as a substitute for a functioning ethical system, and the Proxy Judge is there as a substitute for an exact specification of criteria for true Friendliness.

The simplest conception of a Proxy Judge is a simulated human. It might be a simulation of the SIAI board, or a simulation of someone with a very high IQ, or a simulation of a random healthy adult. A more abstracted version of a Proxy Judge might work without simulation, which is a brute-force way of reproducing human judgment, but I think it would still be all about counterfactual human judgments. (A non-brute-force way of reproducing human judgment is to understand the neural and cognitive processes which produce the judgments in question, so that they may be simulated at a very high level of abstraction, or even so that the final judgment may be inferred in some non-dynamical way.)

The combination of Interim Friendliness and a Proxy Judge is a workaround for incompletely specified strategies for the implementation of Friendliness. For example: Suppose your blueprint for Friendliness is to apply reflective decision theory to the human decision architecture. But it turns out that you don't yet have a precise definition of reflective decision theory, nor can you say exactly what sort of architecture underlies human decision making. (It's probably not maximization of expected utility.) Never fear, here are amended instructions for your developing AI:

With your actions constrained by Interim Friendliness, come up with exact definitions of "reflective decision theory" and "human decision architecture" which meet the approval of the Proxy Judge. Then, apply reflective decision theory (as thus defined) to the human decision architecture (as thus defined), and adopt the resulting decision architecture as your own.

Give the AI a bounded utility function where it automatically shuts down when it hits the upper bound. Then give it a fairly easy goal such as 'deposit 100 USD in this bank account.' Meanwhile, make sure the bank account is not linked to you in any fashion (so the AI doesn't force you to deposit the 100 USD in it yourself, rendering the exercise pointless.)

A variant of Alexandros' AI: attach a brain-scanning device to every person, which frequently uploads copies to the AI's Manager. The AI submits possible actions to the Manager, which checks for approval from the most recently available copy of each person who is relevant-to-the-action.

At startup, and periodically, the definition of being-relevant-to-an-action is determined by querying humanity with possible definitions, and selecting the best approved. If there is no approval-rating above a certain ratio, the AI shuts down.