What sort of evidence would you expect to see if a bureaucracy were a "Strong AI" that you would expect not to see if it were not a "Strong AI"?

I am not asking you to come up with an "avalanche of empirical evidence". I am asking you to distinguish your claim from a floating belief, belief-as-attire, or the like. I am asking what — in anticipated experiences — would it mean for this idea to be true?

This overall topic is known as collective intelligence, where the word "collective" is intended as a contrast to both individual intelligence and AI. There are some folks studying rationality in organizations and management, most notably including Peter Senge who first formulated the idea of a learning organization as a rough equivalent of "rationality" as such.

I strongly suspect that most bureaucracies, to the extent that they exist on a level "higher" than the people who implement them, are dumber than mice.

Cosma Shalizi believes that this is the case. The Singularity (he says) has already happened, not merely in the vague sense that fire/printing/the industrial revolution/computers/whatever have changed things beyond recognition, but in the stronger sense that we are now in the grasp of "vast, inhuman distributed systems of information-processing" with "an implacable drive ... to expand, to entrain more and more of the world within their own sphere", and so on. UFAI (he does not use this term) is here, and we are living in its torture chamber. More here. See footnote 1 of the latter for a brief indication of where he's coming from on this.

A similar trope is present in pop neuro-/ev-psych. The inhuman system of information-processing that imprisons us while duping us into the illusion of freedom is our physical substrate and the evolutionary processes that created it. "We" can do nothing of ourselves, "we" are mere epiphenomenal scum on the surface of forces vaster and more alien than we can comprehend. We are as rats are in the world of humans: we can never amount to anything of real consequence, never have, never will.

It is also present in some forms of Christianity. God and the Devil are warring UFAIs and we are merely pieces in their incomprehensible game. "As flies to wanton boys are we to th' gods, They kill us for their sport."

Which is simply to point out that the idea that we are epiphenomenal scum on the surface of forces vaster and more alien than we can comprehend pops up in multiple contexts. Strangely, though, the alien forces are completely different each time.

I would like to put forth the argument that we already have multiple human-programmed "Strong AI" operating among us, they already exhibits clearly "intelligent", rational, self-modifying goal-seeking behavior, and we should systematically study these entities before engaging in any particularly detailed debates about "designing" AI with particular goals.

What would be the benefit of studying them? Setting aside the question of whether bureaucracies are "Strong AI" (if someone doesn't agree then we can claim that they are "generalized strong AI" or something), what additional information about general laws of intelligence could be gleaned from studying two examples of intelligence rather than only the human one? There are already fields like management or organizational psychology that study the question of designing bureaucracies. Algorithms of corporations are very simple and consist mostly of calls to human black boxes.

A creature's mind can be quite simple even though its limbs are very complex.