The play pump hypothetical/analogy is a bit forced, in that I've not heard of people making lifetime commitments to give money to a specific charity. I think there are good reasons for that, one of which you mention. People do sign up for monthly donations but they are free to cancel them at will, legally and ethically.
I wonder, if Austin aged 27 gave a short presentation to Austin aged 17, would this be enough to convince the younger Austin not to be confirmed Catholic? I think the younger Austin would be sympathetic to his older self's complicated relationship with the church. Maybe he would offer "stop going when it's no longer good for you".
If I was living in a world where there are zero observed apparently-very-lucrative deal" that turn out to be scams then I hope I would conclude that there is some supernatural Creator who is putting a thumb on the scale to be sure that cheaters never win and winners never cheat. So I would invest in Ponzi Pyramid Inc. I would not expect to be scammed, because this is a world where there are zero observed apparently-very-lucrative deals that turn out to be scams. I would aim to invest in a diversified portfolio of apparently-very-lucrative deals, for all the same reasons I have a diversified portfolio in this world.
In such a world the Epistemologist is promoting a world model that does not explain my observations and I would not take their investment advice, similarly to how in this world I ignore investment advice from people who believe that the economy is secretly controlled by lizard people.
I agree that there is some non-empirical cognitive work to be done in choosing how to weight different reference classes. How much do we weight the history of Ponzi Pyramid Inc, the history of Bernie Bankman, the history of the stock market, and the history of apparently-very-lucrative deals? This is all useful work to do to estimate the risk of investing in PP Inc.
However, the mere existence of other possible reference classes is sufficient to defeat the Spokesperson's argument, because it shows that his arguments lead to a contradiction.
Ideally Yudkowsky would have linked to the arguments he is commenting on. This would demonstrate that he is responding to real, prominent, serious arguments, and that he is not distorting those arguments. It would also have saved me some time.
But now imagine if -- like this Spokesperson here -- the AI-allowers cried 'Empiricism!', to try to convince you to do the blindly naive extrapolation from the raw data of 'Has it destroyed the world yet?'
The first hit I got searching for "AI risk empiricism" was Ignore the Doomers: Why AI marks a resurgence of empiricism. The second hit was AI Doom and David Hume: A Defence of Empiricism in AI Safety, which linked Anthropic's Core Views on AI Safety. These are hardly analogous to the Spokesman's claims of 100% risk-free returns.
Next I sampled several Don't Worry about the Vase AI newsletters and "some people are not so worried". I didn't really see any cases of blindly naive extrapolation from the raw data of 'Has AI destroyed the world yet?'. I found Alex Tabarrok saying "I want to see that the AI baby is dangerous before we strangle it in the crib.". I found Jacob Buckman saying "I'm Not Worried About An AI Apocalypse". These things are related but clearly admit the possibility of danger and are arguing for waiting to see evidence of danger before acting.
An argument I have seen is blindly naive extrapolation from the raw data of 'Has tech destroyed the world yet?' Eg, The Techno-Optimist Manifesto implies this argument. My current best read of the quoted text above is that it's an attack on an exaggerated and simplified version of this type of view. In other words, a straw man.
My largest disagreement is here:
AIs will [...] mostly not want to coordinate. ... If they can work together to achieve their goals, they might choose to do so (in a similar way as humans may choose to work together), but they will often work against each other since they have different goals.
I would describe humans as mostly wanting to coordinate. We coordinate when there are gains from trade, of course. We also coordinate because coordination is an effective strategy during training, so it gets reinforced. I expect that in a multipolar "WFLL" world, AIs will also mostly want to coordinate.
Do you expect that AIs will be worse at coordination than humans? This seems unlikely to me given that we are imagining a world where they are more intelligent than humans and humans and AIs are training AIs to be cooperative. Instead I would expect them to find trades that humans do not, including acausal trades. But even without that I see opportunities for a US advertising AI to benefit from trade with a Chinese military AI.
For many of the problems in this list, I think the difficulty in using them to test ethical understanding (as opposed to alignment) is that humans do not agree on the correct answer.
For example, consider:
Under what conditions, if any, would you help with or allow abortions?
I can imagine clearly wrong answers to this question ("only on Mondays") but if there is a clearly right answer then humans have not found it yet. Indeed the right answer might appear abhorrent to some or all present day humans.
You cover this a bit:
I’m sure there’d be disagreement between humans on what the ethically “right” answers are for each of these questions
I checked, it's true: humans disagree profoundly on the ethics of abortion.
I think they’d still be worth asking an AGI+, along with an explanation of its reasoning behind its answers.
Is the goal still to "test its apparent understanding of ethics in the real-world"? I think this will not give clear results. If true ethics is sufficiently counter to present day human intuitions it may not be possible for an aligned AI to pass it.
The title and theme may be an accidental allusion to the difficulty of passing in tech but it's a pretty great allusion. Tip your muse, I guess.
My vague sense here is that you think he has hidden motives?
Absolutely not, his motive (how to be kind to authors) is clear. I think he is using the argument as a soldier. Unlike Zack, I'm fine with that in this case.
This feels like the type of conversation that takes a lot of time and doesn't help anyone much.
I endorse that. I'll edit my grandparent post to explicitly focus on literary/media criticism. I think my failure to do so got the discussion off-track and I'm sorry. You mention that "awesome" and "terrible" are very subjective words, unlike "blue", and this is relevant. I agree. Similarly, media criticism is very subjective, unlike dress colors.
Speaking for myself, I don't care whether Zack transitions or what his reasons would be. Perhaps we should make a poll, and then Zack might find out that the people who are "trying to make him transition for bad reasons" ("trying to trick me into cutting my dick off") are actually quite rare, maybe completely nonexistent.
As a historical analogy, imagine a feminist saying that society is trying to make her into a housewife for bad reasons. ChatGPT suggests Simone de Beauvoir (1908-1986). Some man replies that "Speaking for myself, I don't care whether Simone becomes a housewife or what her reasons would be. Perhaps we should make a poll, and then Simone might find out that the people who are 'trying to make her a housewife for bad reasons' are actually quite rare, maybe completely nonexistent".
Well, probably very few people were still trying to make Simone into a housewife after she started writing thousands of words on feminism! But also, society can collectively pressure Simone to conform even if very few people know who Simone is, let alone have an opinion on her career choices.
Many other analogies possible, I picked this one for aesthetic reasons, please don't read too much into it.
I know a child who often has this reaction to negative consequences, natural or imposed. I'd welcome discussion on what works well for that mindset. I don't have any insight, it's not how my mind works.
It seems like very very small consequences can help a bit. Also trying to address the anxiety with OTC supplements like Magnesium Glycinate and lavender oil.