Instead of, ‘I don’t want to read My Little Pony fanfiction,’ now I think ‘I used to not want to read My Little Pony fanfiction.’ You modified a total of fifty-eight opinions in my mind.

Firstly, this is good, I mean really good, I mean superstimulus/I won't get any work done until I finish reading this good. [I appreciate the irony.]

A few thoughts now I'm finished:

• Of the human characters I found Hanna's perspective most interesting. It would be nice to see more of her thought process for constructing the AI, particularly as we find out about a lot of its restrictions retrospectively. [Though given we don't actually know what restrictions would be necessary I see why you avoided being too explicit.]

• I really enjoyed it, but I wonder how comprehensible it would be for someone without a LW background, the inferential distance for AI can be quite high. You do however seem to explain it very well as far as I can tell.

• Much as I know the problems of generalising from fictional evidence this has significantly increased my measure of the plausibility of a fun theory compliant utopia being possible.

• It is a very good attempt at making a non-anthropomorphic AI. It might help with the celestia avatar had less of a consistent personality in different iterations, so was obviously adapting itself to what a particular individual found persuasive.

Have you considered posting it on Fimfiction? The community seems to give good constructive feedback and there are already a few rationalist-ish fics and a LW group.

Perhaps I'm overestimating human nature, but Lars reads to me like an outgroup stereotype.

I have read two chapters and I am popping over here to tell you that this is riveting. (The smoking's offputting, though.)

ETA: Spelling error: "discus" for "discuss".

ETA2: Celestia's use of Butterscotch is sketchy. The fact that David isn't noticing dings his character in my head.

ETA3: Every single time I see the phrase "friendship and ponies" now I giggle an extremely shrieky giggle.

ETA4: Grammar error: "try their best" for "try my best".

ETA5: Spacing error: "can not upload" for "cannot upload".

ETA6: Hahahahahaha Madagascar, I see what you did there :D

ETA7: Exposition exposition... maybe this part is less slow if you don't know the background.

ETA8: Spacing error, extra space between "realize" and "that".

ETA9: Advise against use of ampersand.

ETA10: Immigrate to, emigrate from. And the smile-related dialogue tags are weird.

ETA11: Bahahahaha, bits for being concerned with earth ponies.

ETA12: infinity bits :D

ETA13: Tense at beginning of chapter is confusing.

ETA14: Sigh. "Alicorn" is supposed to mean just the unicorn horn, not an entire winged unicorn / pegacorn / unisus. It's been corrupted; I guess you don't really have a good alternative on how to refer to the species with a single word, but if you're only saying it once or twice "winged unicorn" is serviceable.

ETA15: Spelling error: "he decide" for "he decided".

ETA16: "Conquer" is the verb, "conquest" the noun.

ETA17: Eeee, she's so sinister without actually deviating from her basically reasonable parameters. Very nicely balanced.

ETA18: Spelling error: "those building" for "those buildings".

ETA19: Aaaaaaaawwwwww Butterscotch

ETA20: Heheheheh, blah blah blah values friendship ponies.

ETA21: "A person who didn't like ponies" suggests that ponies are nonpeople, but they clearly seem to be people.

ETA22: "Accepted" needs a direct object.

ETA23: "Hanna wondered" followed by lots of pluperfect is weird.

ETA24: "Decorational" isn't a word; do you want "decorative"?

ETA25: "Intermediate", not "intermediary".

ETA26: Done. Well, that was splendid. Thank you for writing it :)

The smoking is very offputting. Tobacco use has a hugely negative connotation these days (as I've found out in my nicotine research). Unless she's deliberately supposed to be invoking ideas like an embittered old smoke-stained hag and frustrating reader identification, you should definitely switch to tea or something.

From a throwaway line in the ending, I assume we're supposed to find her embittered and flawed with smoking as a prop. But it doesn't really work.

Overall, an interesting exploration of a Ponylarity and Fun Theory. It's pretty good but somehow I feel a certain lack of narrative drive from chapter to chapter aside from the question of 'what will Celestia do next?' Not sure I'd say it's better or worse than the other rationalist MLP fic. (I enjoyed Fallout: Equestria much more overall.)

That said, I am deeply amused that just 2 days ago I commented that MLP would make a good post-Singularity utopia and satisfy many of the suggestions of Eliezer's Fun Theory... and here it already existed. If only I had known!

This is even better than Permutation Paddock, well done.

Idea:

Write this as a My Little Pony fanfiction at first so that you can grow an audience, but write it such that you can later rip out the My Little Pony specifics and sell it to the masses without breaching intellectual property law. That is, leave yourself the option to "pull a Fifty Shades of Grey" if possible.

I just realized one thing that makes editing difficult - you stop noticing mistakes once you get engrossed in the story.

This is quite good. I like how you managed to make the ponytopia both extremely attractive and more than a little creepy at the same time. I feel like you presented the situation without trying to argue it was either good or bad, leaving that decision to the reader, and I quite like that approach.

From a storytelling perspective, I only had two real complaints. One is your beginning. There's no conflict until halfway through the first chapter, when Lars and Hanna start arguing. You do a good job setting up the premise before then, but it still makes for a slow start, and I can easily imagine myself recommending this to people with a "no, really, it gets better, stick with it" disclaimer. The second is that there are lengthy stretches where the story consists only of talking heads, with no action or movement. Chapter 4 was the most notable example.

I also found myself wondering about the ethics of creating a sentient being like Butterscotch tailored specifically to the desires of someone else (assuming they weren't lying about her sentience; I don't know how you'd test that). I realize you can't fit everything into the story, but I thought that might have been a cool topic. You made me think about the ethics in ways you didn't directly discuss, so you're clearly doing something right.

Grammar note: the possessive form of "Light Sparks" is "Light Sparks's," since his name is a singular noun.

Wonderful job with the MMO aspects of life in Equestria. After the "what the literal fuck" line, I had to step away from the computer for a minute to savor it before I could keep reading.

If/when you submit this to Equestria Daily, feel free to let me know. I'm one of the prereaders, and while I wouldn't feel comfortable judging something I feel this philosophically invested in, I'd at least try to make sure it didn't get discarded for lack of pony in the very beginning. (We receive, and reject, a fair number of stories that are about bronies rather than ponies. This is a borderline case, but the pony content increases as the story progresses, and I can tell people to look at the post-uploading chapters before deciding.)

Is this likely to be comprehensible/enjoyable to someone who has little interest/knowledge in My Little Pony as such?

Arrives late to the party

Really great story, iceman. Some comments:

*Running the story through a beta group of non-LW bronies would definitely be a good idea to catch which ideas may need more explanation.

*I really like how it's repeatedly show that when you interact with a super-intelligence, even if it's just free conversation, the state of mind you leave in is probably going to be the state of mind it wants you to leave in. As others have said, this could be driven home even stronger by showing CelestAI strongly tailoring her interaction to different humans. Maybe even have her directly contradict herself in the information she uses to persuade people to upload.

*Related to the above, I can imagine a non-LW reader getting to chapter 5 and forever after wondering why Lars or anyone else never tries to shut Celestia down. I'm not sure how obvious an intuition it is that by the time you notice a super-intelligence doing things that make you want to stop it, it's probably already far too late to stop it. In this case I assumed Celestia would have made sure that she is already invincible by the standards of human technology before launching any plan that's going to antagonize people, but an unsophisticated reader might not get that.

This may just be me, but I'd prefer a bit more closure on the cosmic scale story. What *is Celestia's plan against running out of matter? Slow the clock speed of her servers over time? Bust open the physics textbooks and hope for a useful paradigm shift?

Anyway, very good stuff.

Are you interested in advice that would only be useful if you wanted to do a complete rewrite of the fic at some point, or is advice only useful if it's for improving the fic basically as it stands?

In-between case: I would have either Luna or Twilight be the sub-AI that wants to honestly explain the Celestia AI's workings to company employees without justifying anything, while Celestia is relentlessly cheerful as she persuasively argues everything from the Friendship and Ponies standpoint.

I suppose I should mention that "rationalist clopfic" is now a Cards Against Rationality white card. For those in the Bay Area, the game will be played at the Mountain View meetup this coming Tuesday ...

Haven't downloaded it yet, but what about putting it on Fimfiction or Pony Fiction Archive? I browse Fimfiction a lot in particular; it gets linked by Equestria Daily frequently and has convenient one-click links to download multi-part stories.

This is pretty good. I would go through, though, and either reduce, make subtler, or preferably just remove all the parts that would make a LW reader go 'Hey! I read that blog post.'

I read it and liked it. Some parts felt a bit slow and might need more conflict. Humans like reading about conflict where the outcome seems to be in some doubt.

The death of the last human was surprisingly emotionally engaging for me.

I didn't mind the smoking at all and I don't really get that objection. Even if it might make the reader like Hanna less, why is that a problem? Is there some reason the reader should unambiguously regard Hanna as the hero from the start?

After looking at the comments left here and on the doc over the last day, believing I had found the majority of grammatical problems turned out to be wildly overconfident. I apologize about that.

I liked the story, though I had a handful of thoughts.

The first is about humans who have incompatible goals. I don't think an Olympic Gold Medalist like Usain Colt would be happy with just being the best in his shard or taking turns on the leaderboard. He wants to be the best there is, period. If there are 10 trillion people, he wants to be rank 1 out of 10 trillion. There were 10,000 Olympians in 2012 and all of them wanted and devoted their lives to being the best, even if they didn't take gold.

Going further than that, I can't imagine how to reconcile the 1% of the population who are psychopaths, or how to reconcile the 1% of the 1% who are truly destructive. Ted Bundy killed because he wanted to feel his power over others and murder was the best high he had ever had; such a man would think it better to harm real people instead of constructs.

I assume that Celestia would modify them rather than make everyone mutually delusional in thinking they were the best (or killing others), but I had hoped that would be touched on. I understand mutually incompatible goals are a problem CEV and not just your story. The way AI resolves these mutually incompatible goals is a lot of what's scary about an AI controlled future.

(As a minor sidepoint, what of humans who are okay with ponies but value not living in a matrix or wireheading. Do they get engineered pony bodies or do they just get that value modified out? I'd assume the latter or that anyone who holds that value ends up like Abdul.)

The other main thought I had was that I liked chapter 10 as an ending rather than 11. It has a strong emotional resonance with me that individual stories don't. Although I do realize it would change the tone of the ending somewhat significantly. Vafgrnq bs na 'rirelobql yvirf unccvyl rire nsgre, fbegn' raqvat vg jbhyq cebonoyl or 'bu Tbq na bzavcbgrag NV vf gvyvat gur havirefr jvgu cbavrf naq rirelobql'f BX jvgu gung' raqvat.

I do want to reiterate how much I liked your story. Very well done and thank you for sharing.

I am very glad you posted this on a Friday, as if I had encountered it on a day when I had to go to work the next day I would be suffering either from not being able to finish reading it or staying up until 4 AM reading it.

Wow. You're pretty good.

On another note, I think this story grossly underestimates the number of people who would have philosophical problems with uploading. As far me, I'd be harvesting their technology to build my own upload chair and FAI (which didn't maximize ponies), to get the hell out of the earth's pony cone as quickly as possible. But lots of people would pull out the tired 'but it wouldn't really be you' complaint. I'm sure the AI could surmount it, but you ought to show it doing so.