Also, whoever saves a person to live another fifty years, it is as if they had saved fifty people to live one more year. Whoever saves someone who very much enjoys life, it is as if they saved many people who are not sure they really want to live. And whoever creates a life that would not have otherwise existed, it is as if they saved someone who had an entire lifetime yet to live.

Robin, we can definitely agree that my notion about relative conditional frequencies is not at all clearly true. This is one of those rare, rare issues that still confuses even me. As such - this is an important general principle, that I'd like to emphasize - when you try to model things that are deeply confusing and mysterious to you, you should not be very confident in your judgments about them.

If infinite people exist, how do our subjective probabilities come out right - why don't we always see every possible die roll with probability 1/6, even when the dice are loaded? How is computation possible, when every if statement always branches both ways? I seriously don't know. Maybe the numbers are finite but just very large. But, if for whatever reason it is possible to flip a biased coin and indeed see mostly heads, then we can try to shape the outcomes of people's lives so that their futures are mostly happy. I don't claim to be sure of this. It is just my attempt to make things add up to normality.

Jeremy, see Nick Bostrom's paper.

In a Big World, which this one appears to be on at least three counts (spatially infinite open universe, inflationary scenario in Standard Model, and Everett branches), everyone who could exist already exists with probability 1. Thus, the issue is not so much creating new people, but ensuring that good things happen to people given that they exist. Creating a new person helps when you can provide them with good outcomes, because what you're really doing is increasing the frequency of good outcomes from that starting point.

Or at least that's one anthropic interpretation of ethics. But it is one reason why I don't endorse running out and creating lots of people if that lowers the average standard of living. In a Big World, it's the average standard of living that you care about.

Eliezer, I hope we can agree that your conclusion is intriguing, but far from clearly true. After all, if every possible person exists, then so does every possible history for every possible person. How then could you effect any relative frequencies?

As for the philanthropist, I think the relevant heuristic is that we approve of anyone who saves lives, to socially reinforce the urge for others to do so. If our instincts developed in a tribal environment, then saving a life, or a small group of lives, is the best that anyone can realistically do, so we had no need to scale our admiration to a larger scale.

But if we are to become less biased, and disdain the philanthropist who spends his life-saving money inefficiently, we should be totally consistent about it, and disdain far, far more those who could spend money to save lives and don't (unfortunately, that probably includes most of us).

Robin: And whoever creates a life that would not have otherwise existed, it is as if they saved someone who had an entire lifetime yet to live.

I have to question that comparison. When you save a life that already exists, you are delivering them from a particular existential danger, even if not from the generic existential danger they face constantly by virtue of being alive. But when you create a life, you are delivering a new "hostage to fortune" and creating an existentially endangered being where none previously existed.

I have a paper on this problem of infinities in ethics: http://www.nickbostrom.com/ethics/infinite.pdf

It is a difficult topic.

Charles, you might want to read some of Peter Singer's writings on this point.

Robin, it's clear that relative frequencies exist and matter somehow, even though it might seem like they shouldn't (e.g. because of the ordering problem described in Dr. Bostrom's paper). We observe random events with nonuniform distributions to occur according to the distribution, as opposed to uniformly. We don't live in an extremely bizarre, acausal world even though there are an infinite number throughout spacetime, because the laws of physics are such as to make bizarre worlds rarer than normal ones (even though there are many more possible bizarre worlds than normal ones). "Difficult topic" is probably an understatement.

Where does this end? If a philanthropist saves one life instead of two he is damned as any murder. Surely we in the more prosperous countries could easily save many lives by cutting back on luxuries, but we choose not to (this would no doubt apply to nearly everyone in these countries) does that make us all murderers?

Robin's comment raises the interesting question of whether creating a new life is as good as saving one. It definitely seems to be easier to create a new one, at least at first (the long term effort is probably greater). Most people manage to create a new life or two, but probably never save any. We don't tend to celebrate new-life creators as much as we do life-savers, perhaps because it is seen as too easy.

how a "duty" gets cashed out computationally, if not as a contribution to expected utility. If I'm not using some kind of moral points, how do I calculate what my "duty" is?

We humans don't seem to act as if we're cashing out an expected utility. Instead we act as if we had a patchwork of lexically distinct moral codes for different situations, and problems come when they overlap.

Since current AI is far from being intelligent, we probably shouldn't see it as compelling argument for how humans do or should behave.

Such questions form the base of some pretty strong theorems showing that consistent preferences must cash out as some kind of expected utility maximization. Sounds right. The more reliable information I get about the world, the more my moral preferences start resembling a utility function. Out of interest, do you have a link to those theorems?

But the consistency assumption is not present in humans, even morally well-rounded ones. We are always learning, intellectually and morally. The moral decisions we make affect our moral values as well as the other way round (this post touched on similar ideas). Seeing morality as a learning process may bring it closer to Paul's queries: what sort of a person am I? What are my values?

Except here the answers to the questions come as a result of the moral action, rather than before it.

Paul, since my background is in AI, it is natural for me to ask how a "duty" gets cashed out computationally, if not as a contribution to expected utility. If I'm not using some kind of moral points, how do I calculate what my "duty" is?

How should I weigh a 10% chance of saving 20 lives against a 90% chance of saving one life?

If saving life takes lexical priority, should I weigh a 1/googleplex (or 1/Graham's Number) chance of saving one life equally with a certainty of making a billion people very unhappy for fifty years?

Such questions form the base of some pretty strong theorems showing that consistent preferences must cash out as some kind of expected utility maximization.

I don't see the relevancy of Mr. Burrows' statement (correct, of course) that "Very wealthy people give less, as a percentage of their wealth and income, than people of much more limited means. For wealthy philanthropists, the value from giving may be in status from the publicity of large gifts."

This is certainly of concern if our goal is to maximize the virtue of rich people. If it is to maximize general welfare, it is of no concern at all. The recipients of charity don't need a percentage's worth of food, but a certain absolute amount.

I'd be curious to know if there is a principled model for optimal human happiness which does not conflict so violently with our moral instincts.

Seems we need to take "creating" and "destroying" humans out of the equation - total or average happiness can work fine in a fixed population (and indeed are the same). We can tweak the conditions maybe, and count the dead and the unborn as having a certain level of happiness - but it will still lead to assumptions that violate our instincts; there will always be moments where creating a new life while making everyone unhappy or killing off someone to raise average happiness will be the right thing for the model to do.

I think we need to deal with "creating" and "destroying" people with other principles than happiness.