I would like to put forth the argument that we already have multiple human-programmed "Strong AI" operating among us, they already exhibits clearly "intelligent", rational, self-modifying goal-seeking behavior, and we should systematically study these entities before engaging in any particularly detailed debates about "designing" AI with particular goals.
They're called "Bureaucracies".
Essentially, a modern bureaucracy - whether it is operating as the decision-making system for a capitalist corporation, a government, a non-profit charity, or a political party, is an artificial intelligence that uses human brains as its basic hardware and firmware, allowing it to "borrow" a lot of human computational algorithms to do its own processing.
The fact that bureaucratic decisions can be traced back to individual human decisions is irrelevant - even within a human or computer AI, a decision can theoretically be traced back to single neurons or subroutines - the fact is that bureaucracies have evolved to guide and exploit human decision-making towards their own ends, often to the detriment of the individual humans that comprise said bureaucracy.
Note that when I say "I would like to put forth the argument", I am at least partially admitting that I'm speaking from hunch, rather than already having a huge collection of empirical data to work from - part of the point of putting this forward is to acknowledge that I'm not yet very good at "avalanche of empirical evidence"-style argument. But I would *greatly* appreciate anyone who suspects that they might be able to demonstrate evidence for or against this idea, presenting said evidence so I can solidify my reasoning.
As a "step 2": assuming the evidence weighs in towards my notion, what would it take to develop a systematic approach to studying bureaucracy from the perspective of AI or even xenosapience, such that bureaucracies could be either "programmed" or communicated with directly by the human agents that comprise them (and ideally by the larger pool of human stakeholders that are forced to interact with them?)
Cosma Shalizi believes that this is the case. The Singularity (he says) has already happened, not merely in the vague sense that fire/printing/the industrial revolution/computers/whatever have changed things beyond recognition, but in the stronger sense that we are now in the grasp of "vast, inhuman distributed systems of information-processing" with "an implacable drive ... to expand, to entrain more and more of the world within their own sphere", and so on. UFAI (he does not use this term) is here, and we are living in its torture chamber. More here. See footnote 1 of the latter for a brief indication of where he's coming from on this.
A similar trope is present in pop neuro-/ev-psych. The inhuman system of information-processing that imprisons us while duping us into the illusion of freedom is our physical substrate and the evolutionary processes that created it. "We" can do nothing of ourselves, "we" are mere epiphenomenal scum on the surface of forces vaster and more alien than we can comprehend. We are as rats are in the world of humans: we can never amount to anything of real consequence, never have, never will.
It is also present in some forms of Christianity. God and the Devil are warring UFAIs and we are merely pieces in their incomprehensible game. "As flies to wanton boys are we to th' gods, They kill us for their sport."
Which is simply to point out that the idea that we are epiphenomenal scum on the surface of forces vaster and more alien than we can comprehend pops up in multiple contexts. Strangely, though, the alien forces are completely different each time.