Most concern about AI comes down to the scariness of goal-oriented behavior. A common response to such concerns is “why would we give an AI goals anyway?” I think there are good reasons to expect goal-oriented behavior, and I’ve been on that side of a lot of arguments. But I don’t think the issue is settled, and it might be possible to get better outcomes without them. I flesh out one possible alternative here, based on the dictum "take the action I would like best" rather than "achieve the outcome I would like best."
(As an experiment I wrote the post on medium, so that it is easier to provide sentence-level feedback, especially feedback on writing or low-level comments.)
What "a single time step" means here depends on what model Arthur learns, which may not be what we intend. For example, suppose a is an action which immediately disables the approval input terminal or the data connection between Arthur and the terminal via a network attack, then taking an arbitrarily long time to secure access to the approval input terminal and giving itself maximum approval. What is approval[T][a] according to Arthur's model?
Overall, don't you think it's too strong to say "But unlike AIXI, Arthur will make no effort to manipulate these judgments." even if Arthur, like short-sighted AIXI, is safer than standard AIXI? As another example, suppose Arthur discovers some sort of flaw in human psychology which lets it manipulate whoever is going to enter the next approval value into giving it maximum approval. Wouldn't Arthur take advantage of that?
I think my description is basically fair, though I might be misunderstanding or just wrong.
There are two ways in which Arthur’s decisions affect approval[T](a); one is by choosing the action a, and the other is by changing the definition of approval[T](a). One-step AIXI cares about both, while Arthur does not. This is what I meant by “Arthur will make no effort to manipulate these judgments.” Note that my proposal requires Hugh to provide “counterfactual” ratings for actions that were not chosen.
Arthur may be motivated to manipulate Hugh’s judgments for other reasons. This is most obvious if Hugh would approve of manipulating Hugh’s judgments, but it could also happen if someone else displaced Hugh and “approved” of actions to manipulate Hugh’s judgments (especially in order to approve of actions that helped them displace Hugh).
In a note on the original document I gave an example to illustrate: suppose that action X causes Hugh to increase all his ratings by 0.2, but that in every case action Y is rated higher than action X by 0.1. Then Arthur will do Y, not X. (Because the expectation of Y is 0.1 higher than the expectation of X, no matter what Arthur believes.)
Even in the one-step takeover case, Arthur doesn’t have an intrinsic incentive to take over. It’s just that he could introduce such an incentive for himself, and so the belief “I will kill Hugh and reward myself if I (kill Hugh and reward myself if I…)” can become a self-fulfilling prophecy. This seems like a different issue than 1 step AIXI's desire to maximize reward.i The main vulnerability is now from external actors.
Incidentally, it also seems relatively easy to prevent these attacks in the case of approval-directed agents. For example, we can make actions too small to encapsulate an entire seizing-control-plan. If you want to use something like this to evaluate plans then you reintroduce the issue, but now in an easier form (since you can watch the plan unfold and take other precautions). I'll write a bit more soon about how I think a reasonable overseer could behave, and in particular on how they can use access to Arthur to improve their evaluations, but for now these are not my largest concerns with the scheme.
(My largest concern is the plausibility of high-level approval-directed behavior emerging from low-level approval-directed behavior. This dynamic may be more brittle than high-level goal-directed behavior emerging from low-level goal-directed behavior.)