I agree that uploading is copying-then-death. I think you're basically correct with your thought experiment, but your worries about vagueness are unfounded. The appropriate question is what counts as death? Consider the following two scenarios: 1. A copy of you is stored on a supercomputer and you're then obliterated in a furnace. 2. A procedure is being performed on your brain, you're awake the entire time, and you remain coherent throughout. In scenario 1 we have a paradigmatic example of death: obliteration in a furnace. In scenario 2 we have a paradigmatic example of surviving an operation without harm. I would say that, if the procedure in 2 involves replacing all or part of your brain, whether it is performed swiftly or slowly is unimportant. Moreover, even if you lost consciousness, it would not be death; people can lose consciousness without any harm coming to them.

Note that you can adjust the first scenario - say, by insisting that the copy is made at the instant of death or that the copying process is destructive as it transpires, or whatever - but the scenario still could go as described. That is, we are supposed to believe that the copy is a continuation of the person despite the possibility of inserting paradigmatic examples of death into the process. This is a clear case of simply stipulating that 'death' (and 'survival') should mean something entirely different. You can't hold that you're still speaking about survival, when you insist on surviving any number of paradigmatic cases of death (such as being obliterated in a furnace). There are no forms of death - as we ordinarily conceive of death - that cannot be inserted into the uploading scenario. So we have here as clear a case of something that cannot count as survival as is possible to have. Anybody who argues otherwise is not arguing for survival, but stipulating a new meaning for the words 'survival', 'death', etc. That's fine, but they're still dead, they're just not 'dead'.

I think this realisation makes understanding something like the brain transplant you describe a little easier. For we can say that we are living so long as we don't undergo anything that would count as dying (which is just to say that we don't die). There's nothing mysterious about this. We don't need to go looking for the one part of the body that maintains our identity under transformation, or start reifying information into a pseudo-soul, or whatever. We just need to ensure whatever we do doesn't count as death (as ordinarily conceived). Now, in the case of undergoing an operation, there are clear guidelines. We need to maintain viability. I cannot do certain things to you and keep you alive, unless I perform certain interventions. So I think the answer is quite simple: I can do anything to you - make any change - as long as I can keep you alive throughout the process. I can replace your whole brain, as long as you remain viable throughout the process, and you'll still be alive at the end of it. You will, of course, be 'brain dead' unless I maintain certain features of your nervous system too. But this isn't mysterious either; it's just that I need to maintain certain structural features of your nervous system to avoid permanent loss of faculties (such as motor control, memory, etc). Replacement with an artificial nervous system is likewise unproblematic, as long as it maintains these important faculties.

A lot of the confusion here comes from unnecessary reification. For example, that the nervous system must be kept structurally intact to maintain certain faculties, does not mean that it somehow 'contains' those faculties. You can replace it at will, so long as you can keep the patient alive. The person is not 'in' the structure (or the material), but the structure is a prerequisite for maintaining certain faculties. The common mistake here is thinking that we must be the structure (or pattern) if we're not the material, but neither claim makes sense. Alternatively, say you have a major part of your brain replaced, and the match is not exact. Somebody might, for example, point out that your personality has changed. Horrified, you might wonder, "Am I still me?" But this question is clearly absurd. There is no sense in which you could ask if you are still you. Nor can you coherently ask, "Did I die on the operating table?" Now, you might ask whether you merely came into existence on the operating table, after the original died, etc. But this, too, is nonsense. It assumes a reified concept of "self" or "identity." There is nothing you can "lose" that would count as a prior version of you 'dying' and your being born anew (whether slightly different or not). Of course, there are such things as irreversible mental degradation, dementia, etc. These are tragic and we rightfully speak of a loss of identity, but there'd be no such tragedy in a bout of dementia. A temporary loss of identity is not a loss of identity followed by gaining a new identity; it's a behavioural aberration. A temporary loss of identity with a change in temperament when one recovers is, likewise, unproblematic in this sense; we undergo changes in temperament regardless. Of course, extreme change can bring with it questions of loss of identity, but this is no more problematic for our scenario than an operation gone wrong. "He never fully recovered from his operation," we might say. Sad, yes, but this type of thing happens even outside of thought experiments.

Upload shmupload.

Lets remove unnecessary complications and consider the more essential question. You are knocked out. While unconscious, a particle-for-particle copy of you is made with all identical energy levels, momenta, spins, colors, flavors, and any other quantum states associated with any of the particles in your body. The only differences are all the particles in the new copy are 3 m to the east of the particles in the original. The unconscious copies are placed someplace nice and revived approximately simultaneously.

Pretty obviously, neither copy feels like it is the other copy. Pretty obviously, each copy presumes it is the original, or presumes it has a 50:50 chance of being the original, but each copy thinks the same as the other, at least initially.

Further, if this process was done to you without your knowledge, and while still unconscious, the original you was destroyed, when the copy came to, it would have no inkling that it was not you and that you were dead. It would think it was you and that you are not dead.

Is there really any important difference between your existence now, and one in which your physical body was replaced by a particle-for-particle copy every so often? To WHOM or WHAT is that difference experienced?

My own conclusion is that the continuity of my life is something of an illusion. And that my distaste for dying is primarily something bred into me by evolution, it is not hard to imagine how it would provide a significant survival advantage over humans without that trait.

I don't know much what to do with this point of view, but one thing I don't do is pay money now to have my severed head frozen after I legally die. It is enough that I carry through on the more straightforward survival instinct type things that evolution has stuck me with.

How do you feel about the continuous uploading procedure described in "Staring into the Singularity"?

It's not the book, it's the story.

Moby Dick is not a single physical manuscript somewhere. If I buy Moby Dick I'm buying one of millions of copies of it that have been printed out over the years. It's still Moby Dick because Moby Dick is the words, characters, events etc. of the story and that is all preserved via copying.

A slight difference with this analogy is that Moby Dick isn't constantly changing as it ages, gaining new memories and whatnot. So imagine that Melville got half way through his epic and then ran out of space in the notebook that I want you to also imagine he was writing it in. So we have a notebook that contains the first half of Moby Dick (presumably, this is a pretty big notebook). Then he finishes it off in a second notebook.

Some time later he pulls a George Lucas and completely changes his mind about where his story was going ("Kill off Ahab? What was I thinking?") and writes a new version of the story where they go into a profitable, if ethically dubious, whaling business with rather more success than in the first version. This is then written up in a third notebook. Now we have three notebooks, the last two of which are both legitimate continuations of the first, carrying on from the exact same point at which the first notebook was ended.

There is no interesting sense in which one of these is some privileged original, as Eliezer puts it. If you can't get your head around that and want to say that, no, the published (in real life) version is the 'real' one imagine that the published version was actually the third notebook. There is no equivalent of publication for identity that could confer 'realness' onto a copy. In real life, neither or them are notebook 1 but they're both continuations of that story.

If Will Riker discovers that he was non-destructively copied by the Transporter and that there's another version of him running around, he will likely think, 'I don't acknowledge this guy as 'me' in any meaningful sense.' The other guy will think the same thing. Neither of them are the same person they were before they stepped into the Transporter. In fact, you are not the same person you were a few seconds ago, either.

Identify yourself as the book and your concept of identity has big problems with or without uploading. Start by reconciling that notion with things like quantum physics or even simple human ageing and you will find enough challenges to be getting on with without bringing future technology into it.

But you are not some collection of particles somewhere. You are the story, not the book. It's just that you are a story that is still in the process of being written. Uploading is no different that putting Moby Dick on a Kindle. If there's still a meat version of you running around then that is also a copy, also divergent from the original. The 'original' is (or was) you as you were when the copy was made.

What's the point of uploading if we have an AI with all the skills and knowledge of everyone not information-theoretically dead at the time of its creation?

I have no idea how to argue with the ideas about consciousness/identity/experience/whatever that make uploading seem like it could qualify as avoiding death. It occurs to me, though, that those same ideas sorta make uploading individuals pointless. If strong AI doesn't happen, why not just upload the most useful bits of people's brainstates and work out how to combine them into some collective that is not one person, but has the knowledge and skills? Why install, say, Yudkowsky.wbe and Immortalbob.wbe, when you could install them as patches to grandfather_of_all_knowledge.mbe, effectively getting (Yudkowsky|Immortalbob)?

Assume that any properties of a brain that can be quantified get quantified, and where the variables match up, the or process takes the maximum. So if brain A is better at math than brain B, brain A|B uses A's math ability. If brain M is male and brain F is female, though, brain M|F will learn from both perspectives (and hopefully be stronger than the originals for it).

So the benefits of a group, with none of the drawbacks, all crammed into one metabrain.

I think the expansion and contraction model, as you've described it, would probably also result in my death. The being that includes the computer and myself would be a new being, of which I am now a component. When the meaty component dies, this is my death, even though there is now a being who perceives itself to be a continuation of me. This being is, in many ways, a continuation of me, just not in the way that I care about most.

I'm not completely sure of this, of course, but anywhere I'm not sure whether I'll die or not I prefer to lean heavily towards not dying.

I think we have to give up on uniqueness of identity in order to remain consistent in these kind of sci-fi scenarios.

edit: And I guess "identity" has to have a continuous value to - similar to the anthropic principle - being x% certain you are in a particular world is like being x% certain you are a particular person.

I don't acknowledge an upload as "me" in any meaningful sense of the term

What about if your mind is uploaded, then downloaded into your clone previously grown without any brain functions? Would you consider the new meat you as "you"?

Suppose that rather than copying my brain, I adjoined it to some external computer in a kind of reverse-Ebborian act; electrically connecting my synapses to a big block of computrons that I can consciously perform I/O to. Over the course of life and improved tech, that block expands until, as a percentage, most of my thought processes are going on in the machine-part of me. Eventually my meat brain dies -- but the silicon part of me lives on.

This is very similar to the premise of Greg Egan's short story, "Learning to be me".

I find this an immensely valuable insight: continuity, or "haecceity", is the critical element of self which naive uploading scenarios dismiss. Our current rational situation of self as concept-in-brain has no need for continuity, which is counterintuitive.

We know a good deal about the universe, but we do not yet know it in its entirety. If there were an observer outside of physics, we might suspect they care great deal about continuity, or their laws might. Depending on your priors, and willingness to accept that current observational techniques cannot access all-that-there-is, it might be worth embedding some value to haecceity near your value of self.

Contrast grow-and-prune uploading with slice-and-scan uploading: the latter will be anathema to the vast majority of humanity; they may "get over it", but it'll be a long battle. And slice-and-scan will probably be much slower to market. Start with Glass and EEGs: we'll get there in our lifetime using grow-and-prune, and our AIs will grow up with mentors they can respect.