A self-modifying AI is built to serve humanity. The builders know, of course, that this is much riskier than it seems, because its success would render their own observations extremely rare. To solve the problem, they direct the AI to create billions of simulated humanities in the hope that this will serve as a Schelling point to them, and make their own universe almost certainly simulated.
Plausible?
If you believe the DA, and you also believe you're being simulated (with some probability), then you should believe to be among the last N% humans in the simulation. So you don't escape the DA entirely.
However, it may be that if you believe yourself to be likely in a simulation, you shouldn't believe the DA at all. The DA assumes you know how many humans lived before you, and that you're not special among them. Both may be false in a simulation of human history: it may not have simulated all the humans and pre-humans who ever lived, and/or you may be in a special subset of humans being simulated with extra fidelity. Not to mention that only periods of your life may be simulated, possibly out of order or without causal structure.
I'm not talking about the DA only, I'm talking about the assumption that our experiences should be more-or-less ordinary. And this is designed to escape the DA; it's the only reason to think you are simulated in the first place.
Really, I got the whole idea from HPMOR: fulfilling a scary prophecy on your own terms.
the assumption that our experiences should be more-or-less ordinary
How do you know what to call "ordinary"? If you think you're being simulated, then you need to predict what kinds and amounts of simulations exist besides the one you're in, as well as how extensive and precise your own simulation is in past time and space, not just in its future.
And this is designed to escape the DA; it's the only reason to think you are simulated in the first place.
There are lots of reasons other than the DA to think we're being simulated: e.g. Bostrom's Simulation Argument (posthumans are likely to run ancestor simulations). The DA is a very weak argument for simulation: it is equally consistent with there being an extinction event in our future.
If I were doing it, I'd save computing power by only simulating the people who would program the AI. I don't think I'm going to do that, so it doesn't apply to me. Eliezer doesn't accept the Doomsday Argument, or at least uses a decision theory that makes it irrelevant, so it wouldn't apply to him.
So - I am still having issues parsing this, and I am persisting because I want to understand the argument, at least. I may or may not agree, but understanding it seems a reasonable goal.
The builders know, of course, that this is much riskier than it seems, because its success would render their own observations extremely rare.
The success of the self-modifying AI would make the builders of that AI's observations extremely rare... why? Because the AI's observations count, and it is presumably many orders of magnitude faster?
For a moment, I will assume I have interpreted that correctly. So? How is this risky, and how would creating billions of simulated humanities change that risk?
I think the argument is that - somehow - the overwhelming number of simulated humanities somehow makes it likely that the original builders are actually a simulation of the original builders running under an AI? How would this make any difference? How would this be expected to "percolate up" thru the stack? Presumably somewhere there is the "original" top level group of researchers still, no? How are they not at risk?
How is it that a builder's observations are ok, the AI's are bad, but the simulated humans running in the AI are suddenly good?
I think, after reading what I have, that this is the same fallacy I talked about in the other thread - the idea that if you find yourself in a rare spot, it must mean something special, and that you can work the probability of that rareness backwards to a conclusion. But I am by no means sure, or even mostly confident, that I am interpreting the proposal correctly.
Anyone want to take a crack at enlightening me?