Moloch's Toolbox (1/2)

Follow-up to: An Equilibrium of No Free Energy


There’s a toolbox of reusable concepts for analyzing systems I would call “inadequate”—the causes of civilizational failure, some of which correspond to local opportunities to do better yourself. I shall, somewhat arbitrarily, sort these concepts into three larger categories:

1.  Decisionmakers who are not beneficiaries;

2.  Asymmetric information;

and above all,

3.  Nash equilibria that aren’t even the best Nash equilibrium, let alone Pareto-optimal.

In other words:

1.  Cases where the decision lies in the hands of people who would gain little personally, or lose out personally, if they did what was necessary to help someone else;

2.  Cases where decision-makers can’t reliably learn the information they need to make decisions, even though someone else has that information; and

3.  Systems that are broken in multiple places so that no one actor can make them better, even though, in principle, some magically coordinated action could move to a new stable state.

I will then play fast and loose with these concepts in order to fit the entire Taxonomy of Failure inside them.

For example, “irrationality in the form of cognitive biases” wouldn’t obviously fit into any of these categories, but I’m going to shove it inside “asymmetric information” via a clever sleight-of-hand. Ready? Here goes:

If nobody can detect a cognitive bias in particular cases, then from our perspective we can’t really call it a “civilizational inadequacy” or “failure to pluck a low-hanging fruit.” We shouldn’t even be able to see it ourselves. So, on the contrary, let’s suppose that you and some other people can indeed detect a cognitive bias that’s screwing up civilizational decisionmaking.

Then why don’t you just walk up to the decision-maker and tell them about the bias? Because they wouldn’t have any way of knowing to trust you rather than the other five hundred people trying to influence their decisions? Well, in that case, you’re holding information that they can’t learn from you! So that’s an “asymmetric information problem,” in much the same way that it’s an asymmetric information problem when you’re trying to sell a used car and you know it doesn’t have any mechanical problems, but you have no way of reliably conveying this knowledge to the buyer because for all they know you could be lying.

That argument is a bit silly, but so is the notion of trying to fit the whole Scroll of Woe into three supercategories. And if I named more than three supercategories, you wouldn’t be able to remember them due to computational limitations (which aren’t on the list anywhere, and I’m not going to add them).

i. For want of docosahexaenoic acids, a baby was lost

My discussion of modest epistemology in Chapter 1 might have given the impression that I think of modesty mostly as a certain set of high-level beliefs: beliefs about how best to combat cognitive bias, about how individual competencies stack up against group-level competencies, and so on. But I predict that many of this book’s readers have high-level beliefs similar to those I outlined in Chapter 2, while employing a reasoning style that is really a special case of modest epistemology; and I think that this reasoning style is causing them substantial harm.

As reasoning styles, modest epistemology and inadequacy analysis depend on a mix of explicit principles and implicit mental habits. In inadequacy analysis, it’s one thing to recognize in the abstract that we live in a world rife with systemic inefficiencies, and quite another to naturally perceive systems that way in daily life. So my goal here won't be to unkindly stick the label “inadequate” to a black box containing the world; it will be to say something about how the relevant systems actually operate.

For our central example, we’ll be using the United States medical system, which is, so far as I know, the most broken system that still works ever recorded in human history. If you were reading about something in 19th-century France which was as broken as US healthcare, you wouldn’t expect to find that it went on working when overloaded with a sufficiently vast amount of money. You would expect it to just not work at all.

In previous years, I would use the case of central-line infections as my go-to example of medical inadequacy. Central-line infections, in the US alone, killed 60,000 patients per year, and infected an additional 200,000 patients at an average treatment cost of $50,000/patient.

Central-line infections were also known to decrease by 50% or more if you enforced a five-item checklist that included items like “wash your hands before touching the line.”

Robin Hanson has old Overcoming Bias blog posts on that untaken, low-hanging fruit. But I discovered while re-Googling in 2015 that wider adoption of hand-washing and similar precautions are now finally beginning to occur, after many years—with an associated 43% nationwide decrease in central-line infections. After partial adoption.1

So my new example is infants suffering liver damage, brain damage, and death in a way that’s even easier to solve, by changing the lipid distribution of parenteral nutrition to match the proportions in breast milk.

Background: Some babies have digestion problems that require direct intravenous feeding. Long ago, somebody created a hospital formula for this intravenous feeding that matched the distribution of “fat,” “protein,” and “carbohydrate” in breast milk.

Just like “protein” comes in different amino acids, some of which the body can’t make on its own and some of which it can, what early doctors used to think of as “fat” actually breaks down into metabolically distinct elements like short-chain triglycerides, medium-chain triglycerides, saturated fat, and omega-6, omega-9, and the famous “omega-3.” “Omega-3” is actually several different lipids in its own right; vegetable oils with “omega-3” usually just contain alpha-linolenic acids, which can only be inefficiently converted to ecosapentaenoic acids, which are then even more inefficiently converted to docosahexaenoic acids, which are the actual key structural components in the body. This conversion pathway is rate-limited by a process that also converts omega-6, so too much omega-6 can prevent you from processing ALA into DHA even if you’re getting ALA.

So what happens if your infant nutrition was initially designed based on the concept of “fat” as a natural category, and all the “fat” in the mix comes from soybean oil?

From a popular book by Jaminet and Jaminet:

Some babies are born with “short bowel syndrome” and need to be given parenteral nutrition, or nutrition delivered intravenously directly to the blood, until their digestive tracts grow and heal. Since 1961, parenteral nutrition has used soybean oil as its source of fat.[6] And for decades, babies on parenteral nutrition have suffered devastating liver and brain damage. The death rate on soybean oil is 30 percent by age four. […]

In a clinical trial, of forty-two babies given fish oil [after they had already developed liver damage on soybean oil], three died and one required a liver transplant; of forty-nine given soybean oil, twelve died and six required a liver transplant.[8] The death-or-liver-transplant rate was reduced from 37 percent with soybean oil to 9 percent with fish oil.2

When Jaminet and Jaminet wrote the above, in 2012, there was a single hospital in the United States that could provide correctly formulated parenteral nutrition, namely the Boston Children’s Hospital; nowhere else. This formulation was illegal to sell across state lines.

A few years after the Boston Children’s Hospital developed their formula—keeping in mind the heap of dead babies continuing to pile up in the meanwhile—there developed a shortage of “certified lipids” (FDA-approved “fat” for adding to parenteral nutrition). For a year or two, the parenteral nutrition contained no fat at all which is worse and can kill adults.

You see, although there’s nothing special about the soybean oil in parenteral nutrition, there was only one US manufacturer approved to add it, and that manufacturer left the market, so…

As of 2015, the state of affairs was as follows: The FDA eventually solved the problem with the shortage of US-certified lipids, by… allowing US hospitals to import parenteral nutrition bags from Europe. And it only took them two years’ worth of dead patients to figure that out!

As of 2016, if your baby has short bowel syndrome, and has already ended up with liver damage, and either you or your doctor is lucky enough to know what’s wrong and how to fix it, your doctor can apply for a special permit to use a non-FDA-approved substance for your child on an emergency basis. After this, you can buy Omegaven and hope that it cures your baby and that there isn’t too much permanent damage and that it’s not already too late.

This is an improvement over the prior situation, where the non-poisonous formulation was illegal to sell across state lines under any circumstances, but it’s still not good by any stretch of the imagination.

Now imagine trying to explain to a visitor from a relatively well-functioning world just why it is that your civilization has killed a bunch of babies and subjected other babies to pointless brain damage.

“It’s not that we’re evil,” you say helplessly, “it’s that… well, you see, it’s not that anyone wanted to kill those babies, it’s just the way the System ended up, somehow…”

ii. Asymmetric information and lemons problems

Three people have gathered in a blank white space:

  • The visitor from a Better World;
  • simplicio , who is attending a major university but hasn’t taken undergraduate economics;
  • cecie, the Conventional Cynical Economist.

The Visitor speaks first.

visitor:  So I’ve listened to you explain about babies suffering death and brain damage from parenteral nutrition built on soybean oil. I have several questions here, but I’ll start with the most obvious one.

cecie:   Go ahead.

visitor:  Why aren’t there riots?

simplicio:  The first thing you have to understand, Visitor, is that the folk in this world are hypocrites, cowards, psychopaths, and sheep.

I mean, I certainly care about the the lives of newborn children. Hearing about their plight certainly makes me want to do something about it. When I see the problem continuing in spite of that, I can only conclude that other people don’t feel the level of moral indignation that I feel when staring at a heap of dead babies.

cecie:  I don’t think that hypothesis is needed, Simplicio. As a start, Visitor, you have to realize that the picture I’ve shown you is not widely known. Maybe 10% of the population, at most, is walking around with the prior belief that the FDA in general is killing people; our government runs on majority rule and the 10% can’t unilaterally defy it.3 Maybe 0.1% of that 10% know that omega-3 ALA is converted into omega-3 DHA via a metabolic pathway that competes with omega-6. And then most of those aren’t aware of what’s happening to babies right now.

visitor:  Pointing to that state of ignorance is hardly a sufficient explanation! If a theater is on fire and only one person knows it, they yell “Fire!” and then more people know it. People from my civilization would scream “Babies are dying over here!” and other people from my civilization would whip around their heads and look.

simplicio:  Our world’s cowards and sheep would hear that and think that it’s (a) somebody else’s problem and (b) all part of the plan.

cecie:  In our world, Visitor, we have an economic phenomenon sometimes called the lemons problem. Suppose you want to sell a used car, and I’m looking for a car to buy. From my perspective, I have to worry that your car might be a “lemon”—that it has a serious mechanical problem that doesn’t appear every time you start the car, and is difficult or impossible to fix. Now, you know that your car isn’t a lemon. But if I ask you, “Hey, is this car a lemon?” and you answer “No,” I can’t trust your answer, because you’re incentivized to answer “No” either way. Hearing you say “No” isn’t much Bayesian evidence. Asymmetric information conditions can persist even in cases where, like an honest seller meeting an honest buyer, both parties have strong incentives for accurate information to be conveyed.

A further problem is that if the fair value of a non-lemon car is $10,000, and the possibility that your car is a lemon causes me to only be willing to pay you $8,000, you might refuse to sell your car. So the honest sellers with reliable cars start to leave the market, which further shifts upward the probability that any given car for sale is a lemon, which makes me less willing to pay for a used car, which incentivizes more honest sellers to leave the market, and so on.

visitor:  What does the lemons problem have to do with your world’s inability to pass around information about dead babies?

cecie:  In our world, there are a lot of people screaming, “Pay attention to this thing I’m indignant about over here!” In fact, there are enough people screaming that there’s an inexploitable market in indignation. The dead-babies problem can’t compete in that market; there’s no free energy left for it to eat, and it doesn’t have an optimal indignation profile. There’s no single individual villain. The business about competing omega-3 and omega-6 metabolic pathways is something that only a fraction of people would understand on a visceral level; and even if those people posted it to their Facebook walls, most of their readers wouldn’t understand and repost, so the dead-babies problem has relatively little virality. Being indignant about this particular thing doesn’t signal your moral superiority to anyone else in particular, so it’s not viscerally enjoyable to engage in the indignation. As for adding a further scream, “But wait, this matter really is important!”, that’s the part subject to the lemons problem. Even people who honestly know about a fixable case of dead babies can’t emit a trustworthy request for attention.

simplicio:  You're saying that people won’t listen even if I sound really indignant about this? That’s an outrage!

cecie:  By this point in our civilization’s development, many honest buyers and sellers have left the indignation market entirely; and what’s left behind is not, on average, good.

visitor:  Your reply contains so many surprising postulates of weird civilizational dysfunction, I hardly know what to ask about next. So instead I’ll try to explain how my world works, and you can explain to me why your world doesn’t work that way.

cecie:  Sounds reasonable.

iii. Academic incentives and beneficiaries

visitor:  To start with, in my world, we have these people called “scientists” who verify claims experimentally, and other people trust the “scientists.” So if our “scientists” say that a certain formula seems to be killing babies, this would provoke general indignation without every single listener needing to study docohexa-whatever acids.

simplicio:  Alas, our so-called scientists are just pawns of the same medical-industrial complex that profits from killing babies.

cecie:  I’m afraid, Visitor, that although there are strong prior reasons to expect too much omega-6 and no omega-3 to be very bad for an infant baby, and there are now a few dozen small-scale studies which seem to match that prediction, this matter hasn’t had the massive study that would begin to produce confident scientific agreement—

visitor:  You’d better not be pointing to that as an exogenous fact that explains your civilization’s problem! See, on my planet, if somebody points to strong prior suspicion combined with confirming pilot studies saying that something is killing innocent babies and is fixable, and the pilot studies are not considered sufficient evidence to settle the issue, our people would do more studies and wouldn’t just go on blindly feeding the babies poison in the meantime. Our scientists would all agree on that!

cecie:  But people loudly agreeing on something, by itself, accomplishes nothing. It’s all well and good for everyone to agree in principle that larger studies ought to be done; but in your world, who actually does the big study, and why do they do it?

visitor:  Two subclasses within the profession of “scientist” are suggesters, whose piloting studies provide the initial suspicions of effects, and replicators whose job it is to confirm the result and nail things down solidly—the exact effect size and so on. When an important suggestive result arises, two replicators step forward to confirm it and nail down the exact conditions for producing it, being forbidden upon their honor to communicate with each other until they submit their findings. If both replicators agree on the particulars, that completes the discovery. The three funding bodies that sustained the suggester and the dual replicators would receive the three places of honor in the announcement. Do I need to explain how part of the function of any civilized society is to appropriately reward those who contribute to the public good?

cecie:  Well, that’s not how things work on Earth. Our world gives almost all the public credit and fame to the discoverer, as the initial suggester is called among us. Our scientists often say that replication is important, but our most prestigious journals won’t publish mere replications; nor do the history books remember them. The outcome is a lot of small studies that have just enough subjects to obtain “statistically significant” results—

visitor:  … What? Probability is quantitative, not qualitative. There’s no such thing as a “significant” or “insignificant” likelihood ratio—

cecie:  Anyway, while it might be good if larger studies were done, the decisionmaker is not the beneficiary—the people who did the extra work of a larger study, and funded the extra work of a larger study, would not receive fame and fortune thereby.

visitor:  I must be missing something basic here. You do have multiple studies, right? When you have multiple bodies of data, you can multiply the likelihood functions from the studies’ respective data to the hypotheses to obtain the meaning of the combined evidence—the likelihood function from all the data to the hypotheses.4

cecie:  I’m afraid you can’t do that on Earth.

visitor:  … Of course you can. It’s a mathematical theorem. You can’t possibly tell me that differs between our universes!

Yes, there are pitfalls for the especially careless. Sometimes studies end up being conducted under different circumstances, with the result that the naively computed likelihood functions don’t have uniform relations to the hypotheses under consideration. In that case, blindly multiplying will give you a likelihood function that’s nearly zero everywhere. But, I mean, if you just look at all the likelihood functions, it’s pretty obvious when some of them are pointing in different directions and then you can investigate that divergence.

Either it makes sense to multiply all the likelihood functions and get out one massive evidential pointer, or else you don’t get a sensible result when you multiply them and then you know something’s wrong with your methods—

cecie:  I’m afraid our scientific community doesn’t run on your world’s statistical methods. You see, during the first half of the twentieth century, it became conventional to measure something called “p-values” which imposed a qualitative distinction between “successful” and “unsuccessful” experiments—

visitor:  That is still not an explanation. Why not change the way you do things?

cecie:  Because somebody who tried using unconventional statistical methods, even if they were better statistical methods, wouldn’t be able to publish their papers in the most prestigious journals. And then they wouldn’t get hired. It’s similar to the way that the most prestigious journals don’t publish mere replications, only discoveries, so people focus on making discoveries instead of replications.

visitor:  Why would anyone pay attention to journals like that?

cecie:  Because university hiring departments care a lot about whether you’ve published in prestigious journals.

visitor:  No, I mean… how did these journals end up prestigious in the first place? Why do university hiring departments pay attention to them?

simplicio:  Why would university hiring departments care about real science? Shouldn’t it be you who has to explain why some lifeless cog of the military-industrial complex would care about anything except grant money?

cecie:  Okay… you’re digging pretty deep here. I think I need to back up and try to explain things on a more basic level.

visitor:  Indeed, I think you should. So far, every time I’ve asked you why someone is acting insane, you’ve claimed that it’s secretly a sane response to someone else acting insane. Where does this process bottom out?

iv. Two-factor markets and signaling equilibria

cecie:  Let me try to identify a first step on which insanity can emerge from non-insanity. Universities pay attention to prestigious journals because of a signaling equilibrium, which, in our taxonomy, is a kind of bad Nash equilibrium that no single actor can defy unilaterally.

In your terms, it involves a sticky, stable equilibrium of everyone acting insane in a way that’s secretly a sane response to everyone else acting insane.

visitor:  Go on.

cecie:  First, let me explain the idea of what Eliezer has nicknamed a “two-factor market.” Two-factor markets are a conceptually simpler case that will help us later understand signaling equilibria.

In our world there’s a crude site for classified ads, called Craigslist. Craigslist doesn’t contain any way of rating users, the way that eBay lets buyers and sellers rate each other, or that Airbnb lets renters and landlords rate each other.

Suppose you wanted to set up a version of Craigslist that let people rate each other. Would you be able to compete with Craigslist?

The answer is that even if this innovation is in fact a good one, competing with Craigslist would be far more difficult than it sounds, because Craigslist is sustained by a two-factor market. The sellers go where there are the most buyers; the buyers go where they expect to find sellers. When you launch your new site, no buyers will want to go there because there are no sellers, and no sellers will want to go there because there are no buyers. Craigslist initially broke into this market by targeting San Francisco particularly, and spending marketing effort to assemble the San Francisco buyers and sellers into the same place. But that would be harder to do for a later startup, because now the people it’s targeting are already using Craigslist.

simplicio:  Those sheep! Just mindlessly doing whatever their incentives tell them to!

cecie:  We can imagine that there’s a better technology than Craigslist, called Danslist, such that everyone using Craigslist would be better off if they all switched to Danslist simultaneously. But if just one buyer or just one seller is the first to go to Danslist, they find an empty parking lot. In conventional cynical economics, we’d say that this is a coordination problem

simplicio:  A coordination problem? What do you mean by that?

cecie:  Backing up a bit: A “Nash equilibrium” is what happens when everyone makes their best move, given that all the other players are making their best moves from that Nash equilibrium—everyone goes to Craigslist, because that’s their individually best move given that everyone else is going to Craigslist. A “Pareto optimum” is any situation where it’s impossible to make every actor better off simultaneously, like “Cooperate/Cooperate” in the Prisoner’s Dilemma—there’s no alternative outcome to Cooperate/Cooperate that makes both agents better off. The Prisoner’s Dilemma is a coordination problem because the sole Nash equilibrium of Defect/Defect isn’t Pareto-optimal; there’s an outcome, Cooperate/Cooperate, that both players prefer, but aren’t reaching.

simplicio:  How stupid of them!

cecie:  No, it’s… ah, never mind. Anyway, the frustrating parts of civilization are the times when you’re stuck in a Nash equilibrium that’s Pareto-inferior to other Nash equilibria. I mean, it’s not surprising that humans have trouble getting to non-Nash optima like “both sides cooperate in the Prisoner’s Dilemma without any other means of enforcement or verification.” What makes an equilibrium inadequate, a fruit that seems to hang tantalizingly low and yet somehow our civilization isn’t plucking, is when there’s a better stable state and we haven’t reached it.

visitor:  Indeed. Moving from bad equilibria to better equilibria is the whole point of having a civilization in the first place.

cecie:  Being stuck in an inferior Nash equilibrium is how I’d describe the frustrating aspect of the two-factor market of buyers and sellers that can’t switch from Craigslist to Danslist. The scenario where everyone is using Danslist would be a stable Nash equilibrium, and a better Nash equilibrium. We just can’t get there from here. There’s no one actor who is behaving foolishly; all the individuals are responding strategically to their incentives. It’s only the larger system that behaves “foolishly.” I’m not aware of a standard term for this situation, so I’ll call it an “inferior equilibrium.”

simplicio:  Why do you care what academics call it? Why not just use the best phrase?

cecie:  The terminology “inferior equilibrium” would be fine if everyone else were already using that terminology. Mostly I want to use the same phrase that everyone else uses, even if it’s not the best phrase.

simplicio:  Regardless, I’m not seeing what the grand obstacle is to people solving these problems by, you know, coordinating. If people would just act in unity, so much could be done!

I feel like you’re placing too much blame on system-level issues, Cecie, when the simpler hypothesis is just that the people in the system are terrible: bad at thinking, bad at caring, bad at coordinating. You claim to be a “cynic,” but your whole world-view sounds rose-tinted to me.

visitor:  Even in my world, Simplicio, coordination isn’t as simple as everyone jumping simultaneously every time one person shouts “Jump!” For coordinated action to be successful, you need to trust the institution that says what the action should be, and a majority of people have to trust that institution, and they have to know that other people trust the institution, so that everyone expects the coordinated action to occur at the critical time, so that it makes sense for them to act too.

That’s why we have policy prediction markets and… there doesn’t seem to be a word in your language for the timed-collective-action-threshold-conditional-commitment… hold on, this cultural translator isn’t making any sense. “Kickstarter”? You have the key concept, but you use it mainly for making video games?

cecie:  I’ll now introduce the concept of a signaling equilibrium.

To paraphrase a commenter on Slate Star Codex: suppose that there’s a magical tower that only people with IQs of at least 100 and some amount of conscientiousness can enter, and this magical tower slices four years off your lifespan. The natural next thing that happens is that employers start to prefer prospective employees who have proved they can enter the tower, and employers offer these employees higher salaries, or even make entering the tower a condition of being employed at all.5

visitor:  Hold on. There must be less expensive ways of testing intelligence and conscientiousness than sacrificing four years of your lifespan to a magical tower.

cecie:  Let’s not go into that right now. For now, just take as an exogenous fact that employers can’t get all of the information they want by other channels.

visitor:  But—

cecie:  Anyway: the natural next thing that happens is that employers start to demand that prospective employees show a certificate saying that they’ve been inside the tower. This makes everyone want to go to the tower, which enables somebody to set up a fence around the tower and charge hundreds of thousands of dollars to let people in.6

visitor:  But—

cecie:  Now, fortunately, after Tower One is established and has been running for a while, somebody tries to set up a competing magical tower, Tower Two, that also drains four years of life but charges less money to enter.

visitor:  … You’re solving the wrong problem.

cecie:  Unfortunately, there’s a subtle way in which this competing Tower Two is hampered by the same kind of lock-in that prevents a jump from Craigslist to Danslist. Initially, all of the smartest people headed to Tower One. Since Tower One had limited room, it started discriminating further among its entrants, only taking the ones that have IQs above the minimum, or who are good at athletics or have rich parents or something. So when Tower Two comes along, the employers still prefer employees from Tower One, which has a more famous reputation. So the smartest people still prefer to apply to Tower One, even though it costs more money. This stabilizes Tower One’s reputation as being the place where the smartest people go.

In other words, the signaling equilibrium is a two-factor market in which the stable point, Tower One, is cemented in place by the individually best choices of two different parts of the system. Employers prefer Tower One because it’s where the smartest people go. Smart employees prefer Tower One because employers will pay them more for going there. If you try dissenting from the system unilaterally, without everyone switching at the same time, then as an employer you end up hiring the less-qualified people from Tower Two, or as an employee, you end up with lower salary offers after you go to Tower Two. So the system is stable as a matter of individual incentives, and stays in place. If you try to set up a cheaper alternative to the whole Tower system, the default thing that happens to you is that people who couldn’t handle the Towers try to go through your new system, and it acquires a reputation for non-prestigious weirdness and incompetence.

visitor:  This all just seems so weird and complicated. I’m skeptical that this scenario with the magical towers could happen in real life.

simplicio:  I agree that trying to build a cheaper Tower Two is solving the wrong problem. The interior of Tower One boasts some truly exquisite architecture and decor. It just makes sense that someone should pay a lot to allow people entry to Tower One. What we really need is for the government to subsidize the entry fees on Tower One, so that more people can fit inside.

cecie:  Consider a simpler example: Velcro is a system for fastening shoes that is, for at least some people and circumstances, better than shoelaces. It’s easier to adjust three separate Velcro straps then it is to keep your shoelaces perfectly adjusted at all loops, it’s faster to do and undo, et cetera, and not everyone is running at high speeds that call for perfectly adjusted running shoes. But when Velcro was introduced, the earliest people to adopt Velcro were those who had the most trouble tying their shoelaces—very young children and the elderly. So Velcro became associated with kids and old people, and thus unforgivably unfashionable, regardless of whether it would have been better than shoelaces in some adult applications as well.

visitor:  I take it you didn’t have the stern and upright leaders, what we call the Serious People, who could set an example by donning Velcro shoes themselves?

simplicio & cecie:  (in unison)  No.

visitor:  I see.

cecie:  Now consider the system of scientific journals that we were originally talking about. Some journals are prestigious. So university hiring committees pay the most attention to publications in that journal. So people with the best, most interesting-looking publications try to send them to that journal. So if a university hiring committee paid an equal amount of attention to publications in lower-prestige journals, they’d end up granting tenure to less prestigious people. Thus, the whole system is a stable equilibrium that nobody can unilaterally defy except at cost to themselves.

visitor:  I’m still skeptical. Doesn’t your parable of the magical tower suggest that, if that’s actually true, somebody ought to rope off the journals too and charge insane amounts of money?

cecie:  Yes, and that’s exactly what happened. Elsevier and a few other profiteers grabbed the most prestigious journals and started jacking up the access costs. They contributed almost nothing—even the peer review and editing was done by unpaid volunteers. Elsevier just charged more and more money and sat back. This is standardly called rent-seeking. In a few cases, the scientists were able to kickstart a coordinated move where the entire editing board would resign, start a new journal, and everybody in the field would submit to the new journal instead. But since our scientists don’t have recognized kickstarting customs, or any software support for them, it isn’t easy to pull that off. Most of the big-name journals that Elsevier has captured are still big names, still getting prestigious submissions, and still capturing big-money rents.

visitor:  Well, I guess I understand why my cultural translator keeps putting air quotes around Earth’s version of “science.” The whole idea of science, as I understand the concept, is that everything has to be in the open for anyone to verify. Science is the part of humanity’s knowledge that everyone can potentially learn about and reproduce themselves. You can’t charge money in order for people to read your experimental results, or you lose the “everyone can access and verify your claims” property that distinguishes science from other kinds of information.

cecie:  Oh, rest assured that scientists aren’t seeing any of this money. It all goes to the third-party journal owners.

simplicio:  And this isn’t just scientists being stupid?

cecie:  No stupider than you are for going to college. It’s hard to beat signaling equilibria—because they’re “multi-factor markets”—which are special cases of coordination problems that create “inferior Nash equilibria”—which are so stuck in place that market controllers can seek rent on the value generated by captive participants.

simplicio:  Weren’t we talking about dead babies at some point?

cecie:  Yes, we were. I was explaining how our system allocated too much credit to discoverers and not enough credit to replicators, and the only socially acceptable statistics couldn’t aggregate small-scale trials in a way regarded as reliable. The Visitor asked me why the system was like that. I pointed to journals that published a particular kind of paper. The Visitor asked me why anyone paid attention to those journals in the first place. I explained about signaling equilibria, and that’s where we are now.

visitor:  I can’t say that I feel enlightened at the end of walking through all that. There must be particular scientists on the editorial boards who choose not to demand replications and who forbid multiplying likelihood ratios. Why are those particular scientists doing the non-sensible thing?

cecie:  Because people in the general field wouldn’t cite nonstandard papers, so if the editors demanded nonstandard papers, the journal’s publication factor would decrease.

visitor:  Why don’t the journal editors start by demanding that paper submitters cite dual replications as well as initial suggestions?

cecie:  Because that would be a weird unconventional demand, which might lead people with high-prestige results to submit those results to other journals instead. Fundamentally, you’re asking why scientists on Earth don’t adopt certain new customs that you think would be for the good of everyone. And the answer is that there’s this big, multi-factor system that nobody can dissent from unilaterally, and that people have a lot of trouble coordinating to change. That’s true even when there are forces like Elsevier that are being blatant about ripping everyone off. Implementing your proposed cultural shift to “suggesters” and “replicators,” or using likelihood functions, would be significantly harder than everyone just simultaneously ceasing to deal with Elsevier, since the case for it would be less obvious and would provoke more disagreement. All that we can manage is to make incremental shifts toward funding more replication and asking more for study preregistration.

To sum up, academic science is embedded in a big enough system with enough separate decisionmakers creating incentives for other decisionmakers that it almost always takes the path of least resistance. The system isn’t in the best Nash equilibrium because nobody has the power to look over the system and choose good Nash equilibria. It’s just in a Nash equilibrium that it wandered into, which includes statistical methods that were invented in the first half of the 20th century and editors not demanding that people cite replications.

visitor:  I see. And that’s why nobody in your world has multiplied the likelihood functions, or done a large-enough single study, or otherwise done whatever it would take to convince whoever needs to be convinced about the effects of feeding infants soybean oil.

cecie:  It’s one of the reasons. A large study would also be very expensive because of extreme paperwork requirements, generated by other systemic failures I haven’t gotten around to talking about yet—7

visitor:  How does anything get done ever, in your world?

cecie:  —and when it comes to funding or carrying out that bigger study, the decisionmaker would not significantly benefit under the current system, which is held in place by coordination problems. And that’s why people who already have a background grasp of lipid metabolic pathways have asymmetric information about what is worth becoming indignant about.

v. Total market failures

visitor:  Even granting the things you’ve said already, I don’t feel like I’ve been told enough to understand why your society is killing babies.

cecie:  Well, no. Not yet. The lack of incentive to do a large-scale convincing study is only one thing that went wrong inside one part of the system. There’s a lot more broken than just that—which is why effective altruists shouldn’t be running out and trying to fund a big replication study for Omegaven, because that by itself wouldn’t fix things.

visitor:  Okay, suppose there had been a large enough study to satisfy your world’s take on “scientists.” What else would likely go wrong after that?

cecie:  Several things. For example, doctors wouldn’t necessarily be aware of the experimental results.

visitor:  Hold on, I think my cultural translator is broken. You used that word “doctor” and my translator spit out a long sequence of words for Examiner plus Diagnostician plus Treatment Planner plus Surgeon plus Outcome Evaluator plus Student Trainer plus Business Manager. Maybe it’s stuck and spitting out the names of all the professions associated with medicine.

cecie:  So, in your world, if there is a dual replication of results on Omegaven versus soybean oil, how does that end up changing the actual patient treatments?

visitor:  By informing the Treatment Planners who specialize in infant ailments that required parenteral nutrition, of course. The discovery would appear inside the “parenteral nutrition” pages in the Earthweb and show up in the feeds of everyone subscribed to that page. The statistics would appear inside the Treatment Planner’s decision-support software. And if all of those broke for some reason, every Treatment Planner for infant ailments that required parenteral nutrition would just use chatrooms. And anyone who ignored the chatrooms would have worse patient outcome ratings, and would lose status relative to Treatment Planners who were more attentive.

cecie:  It sounds like “Treatment Planners” in your world are much more specialized than doctors in this world. I suppose they’re also selected specifically for talent at… cost-benefit analysis and decision theory, or something along those lines? And then they focus their learning on particular diseases for which they are Treatment Planners? And somebody else tracks their outcomes?

visitor:  Of course. I’m… almost afraid to ask, but how do they do it in your world?

cecie:  Your translator wasn’t broken. In our world, “doctors” are supposed to examine patients for symptoms, diagnose especially complicated or obscure ailments using their encyclopedic knowledge and their keen grasp of Bayesian inference, plan the patient’s treatment by weighing the costs and benefits of the latest treatments, execute the treatments using their keen dexterity and reliable stamina, evaluate for themselves how well that went, train students to do it too, and in many cases, also oversee the small business that bills the patients and markets itself. So “doctors” have to be selected for all of those talents simultaneously, and then split their training, experience, and attention between them.

visitor:  Why in the name of—

cecie:  Oh, and before they go to medical school, we usually send them off to get a four-year degree in philosophy first or something, just because.

I don’t know if there’s a standard name for this phenomenon, but we can call it “failure of professional specialization.” It also appears when, for example, a lawyer has to learn calculus in order to graduate college, even though their job doesn’t require any calculus.

visitor:  Why. Why. Why why why—

cecie:  I’m not sure. I suspect the origin has something to do with status—like, a high-status person can do all things at once, so it’s insulting and lowers status to suggest that an esteemed and respectable Doctor should only practice one surgical operation and get very good at it. And once you yourself have spent twelve years being trained under the current system, you won’t be happy about the proposal to replace it with two years of much more specialized training. Once you’ve been through a painful initiation ritual and rationalized its necessity, you’ll hate to see anyone else going through a less painful one. Not to mention that you won’t be happy about the competition against your own human capital, by a cheaper and better form of human capital—and after the sunk cost in pain and time that you endured to build human capital under the old system…

visitor:  Do they not have markets on your planet? Because on my planet, when you manufacture your product in a crazy, elaborate, expensive way that produces an inferior product, someone else will come along and rationalize the process and take away your customers.

cecie:  We have markets, but there’s this unfortunate thing called “regulatory capture,” of which one kind is “occupational licensing.”

As an example, it used to be that chairs were carefully hand-crafted one at the time by carpenters who had to undergo a lengthy apprenticeship, and indeed, they didn’t like it when factories came along staffed by people who specialized in just carving a single kind of arm. But the factory-made chairs were vastly cheaper and most of the people who insisted on sticking to handcrafts soon went out of business.

Now imagine: What if the chair-makers had been extremely respectable—had already possessed very high status? What if their profession had an element of danger? What if they’d managed to frighten everyone about the dangers of improperly made chairs that might dump people on the ground and snap their necks?

visitor:  Okay, yes, we used to have Serious People who would go around and certify the making of some medicines where somebody might be tempted to cheat and use inferior ingredients. But that was before computers and outcome statistics and online ratings.

cecie:  And on our planet, Uber and Lyft are currently fighting it out with taxi companies and their pet regulators after exactly that development. But suppose the whole system was set up before the existence of online ratings. Then the carpenters might have managed to introduce occupational licensing on who could be a carpenter. So if you tried to set up a factory, your factory workers would have needed to go through the traditional carpentry apprenticeship that covered every part of every kind of furniture, before they were legally allowed to come to your factory and specialize in carving just one kind of chair-arm. And then your factory would also need a ton of permits to sell its furniture, and would need to inveigle orders from a handful of resellers who were licensed to buy and resell furniture at a fixed margin. That small, insular group of resellers might not benefit literally personally—in their own personal salary—from buying from your cheaper factory system. And so it would go.

visitor:  But why would the legislators go along with that?

cecie:  Because the carpenters would have a big, concentrated incentive to figure out how to make legislators do it—maybe by hiring very persuasive people, or by subtle bribery, or by not-so-subtle bribery.

Insofar as occupational licensing works to the benefit of professionals at the expense of consumers, occupational licensing represents a kind of regulatory capture, which happens when a few regulatees have a much more concentrated incentive to affect the regulation process. Regulatory capture in turn is a kind of commons problem, since every citizen shares the benefits of non-captured regulation, but no individual citizen has a sufficient incentive to unilaterally spend their life attending to that particular regulatory problem. So occupational licensing is regulatory capture is a commons problem is a coordination problem.

visitor:  Then… the upshot is that it’s impossible for your country to test a functional hospital design in the first place? The reformers can’t win the competition because they’re not legally allowed to try?

cecie:  But of course. Though in this case, if you did manage to set up a test hospital working along more reasonable lines, you still wouldn’t be able to advertise your better results relative to any other hospitals. With just a few isolated exceptions, all of the other hospitals on Earth don’t publish patient outcome statistics in the first place.

visitor:  … But… then—what are they even selling?

simplicio:  Hold on. If you reward the doctors with the highest patient survival rates, won’t they just reject all the patients with poor prognoses?

visitor:  Obviously you don’t evaluate raw survival rates. You have Diagnosticians who estimate prognosis categories and are rated on their predictive accuracy, and Treatment Planners and Surgeons who are rated on their relative outcomes, and you have the outcomes evaluated by a third party, and—

cecie:  In our world, there’s no separation of powers where one person assigns patients a prognosis category and has their prediction record tracked, and another person does their best to treat them and has their treatment record tracked. So hospitals don’t publish any performance statistics, and patients choose the hospital closest to their house that takes their workplace’s insurance, and nobody has any financial incentive to decrease the number of patient deaths from sloppy surgeons or central line infections. When anesthesiologists in particular did happen to start tracking patient outcomes, they adopted some simple monitoring standards and subsequently decreased their fatality rates by a factor of one hundred.8 But that’s just anesthesiologists, not, say, cardiac surgeons.

With cardiac surgeons, a group of researchers recently figured out how to detect when the most senior cardiac surgeons were at conferences, and found that the death rates went down while the most senior cardiac surgeons were away.9 But our scientists have to use special tricks if they want to find out any facts like that.

visitor:  Do your patients not care if they live or die?

cecie:  Robin Hanson has a further thesis about how what people really want from medicine is reassurance rather than statistics. But I’m not sure that hypothesis is necessary to explain this particular aspect of the problem. If no hospital offers statistics, then you have no baseline to compare to if one hospital does start offering statistics. You’d just be looking at an alarming-looking percentage for how many patients die, with no idea of whether that’s a better percentage or a worse percentage. Terrible marketing! Especially compared to that other hospital across town that just smiles at you reassuringly.

No hospital would benefit from being the first to publish statistics, so none of them do.

visitor:  Your world has literally zero market demand for empirical evidence?

cecie:  Not zero, no. But since publishing scary numbers would be bad marketing for most patients, and hospitals are heavily regional, they all go by the majority preference to not hear about the statistics.

visitor:  I confess I’m having some trouble grasping the concept of a market consisting of opaque boxes allegedly containing goods, in which nobody publishes what is inside the boxes.

cecie:  Hospitals don’t publish prices either, in most cases.

visitor:  …

cecie:  Yeah, it’s pretty bad even by Earth standards.

visitor:  You literally don’t have a healthcare market. Nobody knows what outcomes are being sold. Nobody knows what the prices are.

cecie:  I guess we could call that Total Market Failure? As in, things have gone so wrong that there’s literally no supply-demand matching or price-equilibrating mechanism remaining, even though money is still changing hands.

And while I wish that this phenomenon of “you simply don’t have a market” were only relevant to healthcare and not to other facets of our civilization… well, it’s not.

vi. Absence of (meta-)competition

visitor:  I suppose I can imagine imagine a hypothetical world in which one country screws things up as badly as you describe. But your planet has multiple governments, I thought. Or did I misunderstand that? Why wouldn’t patients emigrate to—or just visit—countries that made better hospitals legal?

cecie:  The forces acting on governments with high technology levels are mostly the same between countries, so all the governments of those countries tend to have their medical system screwed up in mostly the same way (not least because they’re imitating each other). Some aspects of dysfunctional insurance and payment policies are special to the US, but even the relatively functional National Health System in Britain still has failure of professional specialization. (Though they at least don’t require doctors to have philosophy degrees.)

visitor:  Is there not one government that would allow a reasonably designed hospital staffed by specialists instead of generalists?

cecie:  It wouldn’t be enough to just have one government’s okay. You’d need some way to initially train your workers, despite none of our world’s medical schools being set up to train them. A majority of legislators won’t benefit personally from deciding to let you try your new hospital in their country. Furthermore, you couldn’t just go around raising money from rich countries for a venture in a poor country, because rich countries have elaborate regulations on who’s allowed to raise money for business ventures through equity sales. The fundamental story is that everything, everywhere, is covered with varying degrees of molasses, and to do any novel thing you have to get around all of the molasses streams simultaneously.

visitor:  So it’s impossible to test a functional hospital design anywhere on the planet?

cecie:  But of course.

visitor:  I must still be missing something. I just don’t understand why all of the people with economics training on your planet can’t go off by themselves and establish their own hospitals. Do you literally have people occupying every square mile of land?

cecie:  … How do I phrase this…

All useful land is already claimed by some national government, in a way that the international order recognizes, whether or not that land is inhabited. No relevant decisionmaker has a personal incentive to allow there to be unclaimed land. Those countries will defend even a very small patch of that claimed land using all of the military force their country has available, and the international order will see you as the aggressor in that case.

visitor:  Can you buy land?

cecie:  You can’t buy the sovereignty on the land. Even if you had a lot of money, any country poor enough and desperate enough to consider your offer might just steal your stuff after you moved in.

Negotiating the right to bring in weapons to defend yourself in this kind of scenario would be even more unthinkable, and would spark international outrage that could prevent you from trading with other countries.

To be clear, it’s not that there’s a global dictator who prevents new countries from popping up; but every potentially useful part of every land is under some system’s control, and all of those systems would refuse you the chance to set up your own alternative system, for very similar reasons.

visitor:  So there’s no way for your planet to try different ways of doing things, anywhere. You literally cannot run experiments about things like this.

cecie:  Why would there be? Who would decide that, and how would they personally benefit?

visitor:  That sounds extremely alarming. I mean, difficulties of adoption are one thing, but not even being able to try new things and see what happens… Shouldn’t everyone on your planet be able to detect at a glance how horrible things have become? Can this type of disaster really stand up to universal agreement that something is wrong?

cecie:  I’m afraid that our civilization doesn’t have a sufficiently stirring and narratively satisfying conception of the valor of “testing things” that our people would be massively alarmed by its impossibility. And now, Visitor, I hope we’ve bottomed out the general concept of why people can’t do things differently—the local system’s equilibrium is broken, and the larger system’s equilibrium makes it impossible to flee the game.

visitor:  Okay, look… despite everything you’ve said so far, I still have some trouble understanding why doctors and parents can’t just not kill the babies. I manage to get up every single morning and successfully not kill any babies. It’s not as hard as it sounds.

cecie:  I worry you’re starting to think like Simplicio. You can’t just not kill babies and expect to get away with it.

simplicio:  I actually agree with Cecie here. The evil people behind the system hate those who defy them by behaving differently; there’s no way they’d countenance anyone departing from the norm. What we really need is a revolution, so we can depose our corrupt overlords, and finally be free to coordinate, and…!

cecie:  There’s no need to add in any evil conspiracy hypotheses here.

It’s sufficient to note that the system is in equilibrium and it has causes for the equilibrium settling there—causes, if not justifications. You can’t go against the system’s default without going against the forces that underpin that default. A doctor who gives a baby a nutrition formula that isn’t FDA-approved will lose their job. A hospital that doesn’t fire that kind of doctor will be sued. A scientist that writes proposals for a big, expensive, definitive study won’t get a grant, and while they were busy writing those failed grant proposals, they’ll have lost their momentum toward tenure. So no, you can’t just try out a competing policy of not killing babies. Not more than once.

visitor:  Have you tried?

cecie:  No.

visitor:  But—

cecie:  Anyway, from my perspective, it’s no surprise if you don’t yet feel like you understand. We’ve only begun to survey the malfunctions of the whole system, which would further include the FDA, and the clinical trials, and the p-hacking. And the way venture capital is structured, and equity-market regulations. And the insurance companies, and the tax code. And the corporations who contract with the insurance companies. And the corporations’ employees. And the politicians. And the voters.

visitor:  … Consider me impressed that your planet managed to reach this level of dysfunction without actually physically bursting into flames.



Next: Moloch's Toolbox part 2.

The full book will be available November 16th. You can go to to pre-order the book, or sign up for notifications about new chapters and other developments.



  1. Carl Shulman notes that the Affordable Care Act linked federal payments to hospitals with reducing central-line infections (source), which was probably a factor in the change. 

  2. Around a thousand infants are born with short bowel syndrome per year in the United States, of whom two-thirds develop parenteral nutrition-associated liver disease (source). See Park, Nespor, and Kerner Jr for a 2011 review of the academic literature, and Koch, Cohen, and Carroll and Madrzyk for news coverage. 

  3. See Tabarrok’s “Assessing the FDA via the Anomaly of Off-Label Drug Prescribing,” which cites the widespread practice of off-label prescription as evidence that the FDA’s efficacy trial requirements are unnecessary. 

  4. See the “Report Likelihoods, Not p-Values” FAQ, or, in dialogue form: “Likelihood Functions, p-Values, and the Replication Crisis.” 

  5. From Schmidt and Hunter’s “Select on Intelligence”: “Intelligence is the major determinant of job performance, and therefore hiring people based on intelligence leads to marked improvements in job performance.” See also psychologist Stuart Ritchie’s discussion of IQ in Vox.

    Software engineer Alyssa Vance adds:

    I’ll note that, as far as I can tell, the informal consensus at least among the best-informed people in software is that hiring has tons of obvious irrationality even when there’s definitely no external cause; see [1] and [2]. In terms of Moloch’s toolbox, the obvious reason for that is that interviewers are rarely judged on the quality of the people they accept, and when they are, certainly aren’t paid more or less based on it. (Never mind the people they reject. “Nobody ever got fired because of the later performance of someone they turned down.”) Their incentive, insofar as they have one, is to hire people who they’d most prefer to be on the same floor with all day long. 

  6. Compare psychiatrist Scott Alexander’s account, in “Against Tulip Subsidies”:

    In America, aspiring doctors do four years of undergrad in whatever area they want (I did Philosophy), then four more years of medical school, for a total of eight years post-high school education. In Ireland, aspiring doctors go straight from high school to medical school and finish after five years. I’ve done medicine in both America and Ireland. The doctors in both countries are about equally good. When Irish doctors take the American standardized tests, they usually do pretty well. Ireland is one of the approximately 100% of First World countries that gets better health outcomes than the United States. There’s no evidence whatsoever that American doctors gain anything from those three extra years of undergrad. And why would they? Why is having a philosophy degree under my belt supposed to make me any better at medicine? […]

    I’ll make another confession. Ireland’s medical school is five years as opposed to America’s four because the Irish spend their first year teaching the basic sciences—biology, organic chemistry, physics, calculus. When I applied to medical school in Ireland, they offered me an accelerated four year program on the grounds that I had surely gotten all of those in my American undergraduate work. I hadn’t. I read some books about them over the summer and did just fine.

    Americans take eight years to become doctors. Irishmen can do it in four, and achieve the same result. Each year of higher education at a good school—let’s say an Ivy, doctors don’t study at Podunk Community College—costs about $50,000. So American medical students are paying an extra $200,000 for…what?

    Remember, a modest amount of the current health care crisis is caused by doctors’ crippling level of debt. Socially responsible doctors often consider less lucrative careers helping the needy, right up until the bill comes due from their education and they realize they have to make a lot of money right now. We took one look at that problem and said “You know, let’s make doctors pay an extra $200,000 for no reason.”

    For a more general discussion of the evidence that college is chiefly a costly signal of pre-existing ability, rather than a mechanism for building skills and improving productivity, see Bryan Caplan’s argument in “Is College Worth It?”, also summarized by Roger Barris

  7. See, e.g., Scott Alexander’s “My IRB Nightmare.” 

  8. From Hyman and Silver, “You Get What You Pay For”:

    By the 1950s, death rates ranged between 1 and 10 per 10,000 encounters. Anesthesia mortality stabilized at this rate for more than two decades. Mortality and morbidity rates fell again after a 1978 article reframed the issue of anesthesia safety as one of human factor analysis. In the mid-1980s, the American Society of Anesthesiologists (ASA) promulgated standards of optimal anesthesia practice that relied heavily on systems-based approaches for preventing errors. Because patients frequently sued anesthetists when bad outcomes occurred and because deviations from the ASA guidelines made the imposition of liability much more likely, anesthetists had substantial incentives to comply.

    [… W]e should consider why anesthesia mortality stabilized at a rate more than one hundred times higher than its current level for more than two decades. The problem was not lack of information. To the contrary, anesthesia safety was studied extensively during the period. A better hypothesis is that anesthetists grew accustomed to a mortality rate that was exemplary by health care standards, but that was still higher than it should have been. From a psychological perspective, this low frequency encouraged anesthetists to treat each bad outcome as a tragic but unforeseen and unpreventable event. Indeed, anesthetists likely viewed each individual bad outcome as the manifestation of an irreducible baseline rate of medical mishap.

    Hyman and Silver note other possible factors behind the large change, e.g., the fact that the person responsible for mishaps was often easy to identify since there tended to be only one anesthetist per procedure, and that “because surgical patients had no on-going relationships with their anesthetist, victims were particularly likely to sue.” 

  9. See Jena, Prasad, Goldman, and Romley, “Mortality and Treatment Patterns Among Patients Hospitalized With Acute Cardiovascular Conditions During Dates of National Cardiology Meetings.” 


sorted by
magical algorithm
Highlighting new comments since Today at 10:39 PM
Select new highlight date
All comments loaded

Simplicio is a weak man.

Cecie's model seems like it would have difficulty explaining things like the replication crisis, in which a large group of scientists coordinated to mount a massive challenge to bad norms, on their own time, before a major funder stepped in.

But a more charitably portrayed Simplicio would have no trouble explaining it. These scientists were educated to believe in the idea of collaborative knowledge creation through experimental testing, had the courage of their convictions, were willing to talk to each other about problems, were willing to follow virtuous leadership, and finally had enough people willing to serve as such leaders to initiate a response.

In other words, they are neither hypocrites, nor cowards, nor psychopaths, nor sheep, but a free people. As free citizens of the republic of science, they don't have to mindlessly do whatever their incentives tell them to.

Understanding where and how this happens, and what things make it more or less likely, seems to me like the single most important social science question right now. To see what the grand obstacle is to people solving these problems by, you know, coordinating.

Sure, there exist local incentives to participate in these unfortunate situations. But, the existence of a bad rather than a good equilibrium is itself something that needs to be explained. In particular, it seems like in a lot of situations, people who break with the system are viewed as fools who don't know how to play the game, where at other times and places they'd have been viewed as heroes displaying good citizenship and leading by example. I don't think it's unreasonable to describe this as people being bad (because we've been taught to be). There has been a massive campaign to train people to behave as price-takers rather than as free citizens.

The incentives model doesn’t do well on how children learn, how hoplite phalanxes fight, how primitive communism works, how righteousness works, how conspiracies and governments and law enforcement work, and in general how there come to be the background social structures in which microeconomic incentives can play out.

Primitive communism isn't people nobly resisting the impulse to hoard or shirk - it's just, people doing the sensible thing, since everyone depends on the shared project, so you'd like to see it succeed, so if something needs doing, you do it.

Children don't do as little learning as they can get away with, at least until school trains them to. They make strenuous, spontaneous, persistent efforts to learn. And not just things for calculated personal advantage - they work hard to learn the customs of their family, tribe, and nation, and how to contribute.

And so on.

No hospital would benefit from being the first to publish statistics, so none of them do.

Hospital statistics have been published for several years now on Hospital compare. Similar programs exist for outpatient and nursing home quality metrics. This is largely due to effort by the Obama administration.

(Of course, this might just shift the "inadequate equilibrium" question to: if these statistics are published, how come so few people use them?)

Do you know if they normalize for case difficulty? If a hospital patients seems like it will get worse outcomes.

That's a great question and yes, there is generally some amount of risk adjustment performed. To take the example Eliezer used early in this post: you can find much more information then you probably wanted about the risk adjustment for central line associated bloodstream infections (CLABSI) here .

It just occurred to me that your link is likely to be an Incredibly Important(tm) tool for the weird sort of person who might actually be interested in themselves/friends/family not dying during a procedure.

(In opposed to just being interested in signaling how caring we are, or seeking medicine to feel cared for.)

Practical, actionable advice ideas:

Seriously, if bookmarking and remembering a tab can change your life expectency by years, that's one hell of a cost/benefit ratio. I'd put absurdly high odds on it being worth picking out a default hospital beforehand, so you don't have to make a split-second decision in an emergency. Like, you just never see low hanging fruit with cost/benefit ratios that high in day-to-day life.

Like, maybe print off the info for a couple choices, and magnet them to the fridge with the closest one on top and the best one underneath? Add closest address to your GPS, and drive the route once so you know that you know it? I'm pretty sure this could be optimized further with more thought.

Also, I can't find it now, but I think I recall Robin Hanson commenting on proximity of home to the nearest hospital being a major determinent in life expectency, along with things like rural living. I think someone was musing about whether property values were any higher near hospitals, but I don't remember if they got an answer or what it was.

I just did this and it was pretty easy! And in fact I decided to change the hospital I go to by default.

If you pick out your default hospital it might be worthwhile to share the information on facebook with a nice hashtag.

When I was hit by a car in third grade, my mother insisted that they not take me to the local hospital 3min away but rather to the one 40min away, due to inside knowledge from being in nursing training. Impossible to say that it would've been bad if I'd gone to the local one, but ending up at the "good" one certainly didn't hurt.

So on the object level, if I ever happen to have a baby who needs parenteral nutrition, is there some nutty religious belief I can claim to have that will let me put fish oil in it or a way to demand transfer to Boston Children's Hospital or something?

More generally, could we fight all such problems of this class by claiming to believe in Moloch, as a vengeful god? Then ask for religious exemption from all coordination problems where exemption is legally possible.

Why not organize a religion to spite its god, rather than worship it?

EDIT: The concrete benefits could come from a single commandment to defy Moloch whenever possible. All shoes must be velcro and exempt from dress codes, doctors must be specialists when available, and you can sue for religious discrimination if someone makes hiring decisions based on autodidact/community college attendance over Ivy League, etc.

I know the IRS's definition of a religion is deliberately fuzzy, and allows anyone with a "sincerely held belief" to call their thing a religion. If other religious exemptions are similarly open, then it would be hilarious way to fight Moloch. After all, coordination problems are a real, empirically verifiable thing, which economists at least sincerely believe in.

If the Flying Spaghetti Monster is considered a valid belief in some legal contexts, then so should Moloch. And, personifications of natural forces were some of the first historical gods, so there's a precedent. Egregores may not be physical things, but believing in the processes they are names for should qualify if people believing in separate "Non-Overlapping Magisteria" qualifies.

There's a big gap between "should work" and "works in practice", though. Anyone know how big this particular gap is?

Well, the velcro-shoes solution is mostly a societal expectation that a legal expectation, so that wouldn't be solved.

It would give you a weapon in some contexts, like the workplace; if anyone made fun of your shoes, you could claim religious discrimination. That might be evil though.

The shoes I'm currently wearing have neither velcro nor shoelaces.

They use elastic bands and manage to look like normal shoes that nobody will notice as anything besides normal black leather shoes.

My predcition is that having a sincerely held belief to 'defy Moloch whenever possible' would result in suffering the harm caused by being the first actors to switch from the worse Nash equilibrium.

Let's talk about how timed-collective-action-threshold-conditional-commitment.

That sounds crazy, but I like it. I expect that even if it doesn't actually fix things, it will at least give some attention to the problem.

I love this post - it really helps to have specific and clear explanations of what causes institutions to be broken (I plan to go through this later with a friend to internalise the main mechanisms). Also I found the dialogue format fun, which cut against the length of the post massively (i.e. I didn't mind reading something as long because I enjoyed it). Anyhow, promoted to Featured.

Two things: an expansion on the "employers optimizing for IQ" model, and a defense of regulations as critical tools for *solving* coordination problems.

suppose that there’s a magical tower that only people with IQs of at least 100 and some amount of conscientiousness can enter, and this magical tower slices four years off your lifespan. The natural next thing that happens is that employers start to prefer prospective employees who have proved they can enter the tower.

I think most companies are sufficiently broken that they aren't even capable of optimizing for the metrics which would earn them the most money. (Before anyone cries “EMH”, It’s a Principal-Agent Problem, and probably unexploitable unless you unilaterally control a whole company. :p)

For example, you'd think that law firms would be as ruthlessly motivated by money as anyone. They should always use the best available metrics to hire the most cost-effective lawyers. But, as Robin Hanson points out, they ignore track records and hire based on fuzzy personal impressions, even when this leads to demonstrably worse outcomes. And, this is pattern repeats in media pundits, teachers, etc. Why? What on earth are we actually optimizing for, if not expected revenue generation by the new hire?

Robin's answer is that hiring managers are optimizing for looking good to their bosses, and to powerful elites more generally. In short, hiring managers are optimizing for prestige in the eyes of everyone who's opinion they care about, not money. (At least, not money for the firm.) Maybe they do care about IQ to the degree that it gives them prestige, but not as much as you would expect from assuming they're maximizing expected profits.

We like to see ourselves as egalitarian, resisting any overt dominance by our supposed betters. But in fact, unconsciously, we have elites and we bow to them. We give lip service to rebelling against them, and they pretend to be beaten back. But in fact we constantly watch out for any actions of ours that might seem to threaten elites, and we avoid them like the plague. Which explains our instinctive aversion to objective metrics in people choice, when such metrics compete with elite advice.

Ok, onto my second point:

I absolutely love the extended Tower metaphor for College burning 4 years of life and money on runaway signaling competitions. (Although it’s never stated explicitly, it screams it once the thought strikes you, and footnote 6 corroborates this. Not sure if footnote 5 was supposed to include a link to the original SSC comment, so I can’t verify that the tower metaphor started as a college metaphor.) At least, I loved everything but this:

simplicio: I agree that trying to build a cheaper Tower Two is solving the wrong problem. The interior of Tower One boasts some truly exquisite architecture and decor. It just makes sense that someone should pay a lot to allow people entry to Tower One. What we really need is for the government to subsidize the entry fees on Tower One, so that more people can fit inside.

That's a bit of an unfair straw-man of tuition subsidies, conditional on the college metaphor being intentional. If we wanted to read people bashing the outgroup, we'd all go to /r/atheism. Like, I got a strong "childish kicking someone when they can't kick back" reaction, which I found hard to put aside to read the rest. It's thinly veiled, so not all liberals will realize they’re being kicked, but that doesn't help much.

Obligatory defense which I don’t want to give but feel obligated to anyway: tuition subsidies are mostly for community colleges, or are only large enough to cover just what community college would have cost if the student decides to burn the cash on the added prestige of another school. No one is suggesting everyone attend Harvard, or hand out Harvard-sized subsidies.

And, college does offer some real education along with the signaling. The art degrees are a form of countersignaling by the upper class, but if you look at the degrees that lower class people get, I would bet that they skew much more toward trade schools and more practical information. (Also, observation-selection effect may be incredibly strong here.) So, tuition assistance programs likely do far more good than harm. Sure, it would help more if some of it wasn't going to zero-sum signaling competitions, but the subsidies themselves are hardly as big a driver as they would be in the Tower model.

</end obligatory defense of defenseless punching bag.>

And, more generally, I see this as a symptom of a larger problem. I Agree Denotationally But Object Connotationally with a lot of the libertarian undercurrents.

Maybe future chapters will feture more of the Dilbert-scale inadequacies in industry, like Robin Hanson's point on law firms. But I'm worried not, based on quips like this:

visitor: Do they not have markets on your planet? Because on my planet, when you manufacture your product in a crazy, elaborate, expensive way that produces an inferior product, someone else will come along and rationalize the process and take away your customers.

Although, while scanning through looking for that quote, I noticed the Craigslist thing, which removed about half my worry. The remainder would be solved by explicitly stating that privatizing and/or deregulating everything won't magically make everything better, as they appear to have on the Visitor’s planet.

(And, for those who need evidence for that claim:

1) Cost Disease is just as bad in for-profit hospitals as not-for-profit ones.

2) SSC on Rehab clinics:

They’re minimally regulated. There’s no credentialing process or anything. There are many different kinds, each privately led, and low entry costs to creating a new one. They can be very profitable – pretty much any rehab will cost thousands of dollars, and the big-name ones cost much more. This should be a perfect setup for a hundred different models blooming, experimenting, and then selecting for excellence as consumers drift towards the most effective centers. Instead, we get rampant abuse, charlatanry, and uselessness.
On the other hand, when the government rode in on a white horse to try to fix things, all they did was take the one effective treatment [which the rehab clinics also strongly discourage], regulate it practically out of existence, then ride right back out again. So I would be ashamed to be taking either the market’s or the state’s side here."

So, it looks to me that any given regulation has about 1:1 odds of either hurting or helping. I would love to have a density-map of inadequate equilibria by things like different industries, regions, scientific/academic disciplines, section of government, type of organization (publicly traded, privately owned, government owned, charity, church, co-op, hobby, club, sport, and anything else humans do in groups), size of organization, age of organization, organizational structure, etc. Not as a way to choose designs, since honestly most human institutions need to be redesigned from the ground up using game theory, but as a map of what needs fixing.)

So, using all these politicized examples is probably a Bad Idea.This is difficult to avoid, given the subject mater, but your usual habit of using historical examples whenever making inherently political points was a good one. Maybe that could be done for the most controversial ones? (Doing it for all would leave the reader thinking "yeah, but is that even still a problem today?")

And, I especially don't want a “yay-markets! Boo-government!” message to dissuade EAs from interacting with politicians or trying to push for legislation. It can be extremely influential to pull sideways in policy tug-of-war. Both parties may be pulling in opposite directions on some issues, and have no spare time (read: no free energy in the system) to invest in researching possible policies on an orthogonal axis. If you do all that for them though, and just hand them a Pareto-improvement, then both sides will likely support it, or at least not oppose it bitterly with every fiber of their being, like with partisan issues.

After all, what politician wants to be the guy to kill a important-sounding thing with no downsides? They might support the regulation or whatever it is, just to avoid the bad press.

Government's main benefit to humanity is the ability to unilaterally solve coordination problems. It can unilaterally discourage pollutants, or build roads, or regulate monopolistic utilities. Sure, things like unions can fight monopolies too, but nothing else has quite the versatility of government. It's not the only tool we have in the fight against Moloch, but it's the biggest single one. Maybe Moloch is corrupting it too, but it hasn't fully lost that battle yet.

The visitor definitely seems to be from dath ilan, so I guess this is looking at how we can get there.

(Formatting note: Add a space after adding a markdown link to actually transform the markdown into a link)

Firstly, I've heard two-factor markets called the Chicken and the Egg Problem in startup blogs. You need the Chicken before the Egg and the Egg before the Chicken, so there seems to be no way to get it off the ground.

Signal Equilibria seems very related to the recent article on Signal Seeding.

Decisionmakers who are not beneficiaries

This is just the principal-agent problem, right? (Which is a topic that’s been well-studied; see the linked Wikipedia page.)

I recall reading an article about how some Indian hospitals practiced much greater surgical specialization than US hospitals with the positive results you'd expect so I don't think the international situation is quite as hopeless as claimed here. But on the other hand Certificates of Need are a thing in the US that explicitly prevent competition between hospitals.


The term you're looking for here is assurance contract. (And yes, they're horribly underused.)

The link is broken, I think + Didn't Alex Tabarrok do one better by creating the dominant assurance contract?

Blargh, thanks. Stupid pseudo-Markdown rich text editor... this thing is really awful. We really just need a straightforward comment box, ideally supporting preview for Markdown stuff, with optionally some sort of rich-text editor for those who want it; this current system is just an incomprehensible, uncontrollable mess.

"I'm not aware of a standard term for this situation, so I'll call it an inferior equilibrium"

Lock In! I think what you're describing is called Lock In. Classic example being the qwerty keyboard. Or not using the metric system yet. Path-dependence of arriving at a particular convention, where other better alternatives exist and are known and available, but there are costs of switching.

Nice catch!

Googling the term brought me to the Vendor Lock-In Wikipedia page, but no page just for "lock in", even though that's a common term for this sort of thing. However, the "see also" section mentions Path Dependence, which mentions the Bandwagon Effect, which is the perfect term for the Craigslist phenomenon.

These aren't all quite the same thing, but they all seem related. They all highlight different aspects or special cases of similar phenomena.

Your view of the medical system suggests that the government regulations prevent all alternative medicine from existing. That's not true.

Individual citizens spend money to buy medical products outside of the official system. In Germany a "Heilpraktiker" can perform many different medical interventions. To become a Heilpraktiker a person has to pass a reasonable written and oral test and afterwards they can engage in diagnozing and curing illnesses. They can't get the insurance companies to pay for a given treatment, they can't treat infectious diseases (you don't want those diseases to spread), a few lobbies like dentists succeeded in preventing Heilpraktikers from operating on their turff but they can do a lot and can even operate.

Hospital are big institutions and building a better hospital takes a lot of energy but many medical issues can be treated on a much smaller scale.

I see the appeal of wanting to separate the task of medicine among multiple specialists but I don't think that would be needed to get change. If you have multiple treatment provides who tell you predictions about the likely outcome of getting treated by them you can have an effective market in a way that needs less complexity.

I think you would likely find a jurisdiction where you could start a company that manages a market where medical intervention could be brought with open prices and open knowledge about likely outcomes through the mechanism I described in my post about prediction-based medicine.

"It’s hard to beat signaling equilibria—because they’re “multi-factor markets”—which are special cases of coordination problems that create “inferior Nash equilibria”—which are so stuck in place that market controllers can seek rent on the value generated by captive participants."

It's things like this that makes me puzzled why the author is yet libertarian, even in spite of this perfectly succint description of the Awful state of affairs arrived at in the wild west, when a Leviathan could try to transcend/straddle these fruits/niches and force them upward into a more Pareto optimal condition, maybe even into the non-Nash E. states if we're extra lucky. Or is Regulatory Capture and the Law of Unintended Consequences generally so strong that any attempt can be assumed futile, and we'll always achieve more harm than good in some lateral unexpected way as the market circumvents our efforts? (The Germans have a wonderful word - Verschlimmbessern! When you try to help/improve something, with good intentions but end up breaking it worse) Is it only thus today because all regulators also come with the undesideratum of human error? Would an AI too be doomed & unable to remedy these inferior equilibria? Or does the libertarian view only apply pragmatically now at the human era.

a Leviathan could try to transcend/straddle these fruits/niches and force them upward into a more Pareto optimal condition, maybe even into the non-Nash E. states if we're extra lucky.

Remember that old Yudkowsky post about Occam's Razor, wherein he points out how "a witch did it" sounds super-simple, but the word "witch" hides a ton of hidden complexity? I'm pretty sure you're doing the same thing here with the word "could". Instead of trying to picture what an imaginary all-powerful leader could do, imagine what a typical leader would do. You've just given a speech to Hillary Clinton about why it's imperative to decimate the university system for economic reasons, or to Donald Trump about how protectionism is always Pareto-suboptimal. Do they react the way you wish them to act? Does their reaction make you want to give them more forceful power, or less?

For that matter, you don't have to imagine any speeches - every politician and their staff have access to the same internet you do, and can learn everything about coordination problems and their solutions that you can, and even without totalitarian official power they still have enough of a "bully pulpit" and enough credibility, seriousness, and so on to push the world out of at least the shallowest non-globally-optimal local optima. How many of them are doing so? By contrast, how many of them are deliberately setting up more barriers to such changes?

He's not that libertarian in the political sense, though probably more than either of us.

I don't think that using a label like libertarian in this was is helpful for discussing the problem. Identity-labels hinder clear thinking about politics. EY doesn't engage into saying that he's a libertarian in a post like this.

On doctors:

While i do see a benefit of isolating diagnostics from treatment if you started evaluating outcomes, done on its own it seems like a bad idea:

To diagnose well you need to know which things are relevant for treatment, which means learning most of same knowledge as the treater would have.

Information transfer between humans is time-taking and lossy, so the Treater would most likely have much less knowledge to base his treatment on than the Diagnoser, leading to increased risks of wrong treatment (for a practical example if the initial diagnosis was wrong, original diagnoser would be in a much better position to notice this when getting more information during treatment than a Treater who did not see the original data).

Also teaching quality will be much worse if teachers are different people than those actually doing the work, a teacher who works with what he is teaching gets hours of feedback everyday on what works and what does not, a teacher who only teaches has no similar mechanism, so he will provide much less value to his students.

Also teaching quality will be much worse if teachers are different people than those actually doing the work, a teacher who works with what he is teaching gets hours of feedback everyday on what works and what does not, a teacher who only teaches has no similar mechanism, so he will provide much less value to his students.

No objectsion to the rest of your post, but I'm with Elizer on this. Teaching is a skill that is entirely separate from whatever subject you are teaching and this skill also strongly influences the amount of value a teacher can provide to their students. If you combine the tasks you end up selecting/training for two separate skillsets, which means you get people that are ill optimized for at least one of their tasks.

Maybe we can have the healer-doctors oversee the curriculum taught by the teacher-doctors?

There are different ways to treat a given problem. There are two ways to deal with myome's in the uterus. One is radiation and the other is surgery. Both of those treatments require very different skills. If you let the people who make their money with selling surgery do the diagnosis you get a lot more removed uterus than if you let a person who doesn't have a stake in whether the treatment is surgery or radiation make the decision.

A few years ago I saw a presentation by a person at a Health 2.0 event which basically made the case that as a result of different organization of doctors in France and Germany and the resulting incentives, the doctors in France do a lot more of the radiation instead surgery and as a result a lot more French woman keep their uterus.

The whole presentation wasn't public and publically the person holding the presentation was running a website telling woman about myome treatment that didn't include any section on shaming doctors, because she didn't want to alinate potential stake-holders in her quest to reduce the number of operations.

Surgery salesmen shouldn't be the people who do the initial diagnosis and tell the person what treatment to get.

In the BFR announcment Musk promized that tickets from intercontinental rocket travel for the price of a current economy ticket. If that's the price to travel by rocket it would also be the price to go to a space station.

Maybe there are some illnesses where you can benefit from being in zero g and you can have your hospital outside of any jurisdiction.

In the BFR announcment Musk promized that tickets from intercontinental rocket travel for the price of a current economy ticket.

"Full-fare" economy, which is much more expensive than even the "typical" international economy seat tickets you're thinking of, but yes, and even outsiders don't think it's impossible. It is very sensitive to a lot of assumptions - third party spreadsheets I've seen say low thousands of dollars per ticket is possible, but it wouldn't take many assumptions to fall short before prices jumped into the low tens of thousands.

If that's the price to travel by rocket it would also be the price to go to a space station.

No; a space station trip could even be an order of magnitude more expensive. (Which I admit surprised me too).

An intercontinental ballistic arc can require nearly as much fuel as Low Earth Orbit, true, but some hops are shorter than others, and in every case the difference then goes through an exponential function in the rocket equation. BFR might end up needing half as much propellant on a mere transatlantic flight, and the whole "super cheap flights" plan is predicated on them reducing non-propellant costs to a small multiple of propellant costs, so that's not a trivial change.

Also, an intercontinental flight can pack people in like sardines, with enough life support to go up and down again. A flight to a space station implicitly includes the cost of lifting and operating (and if it's going to be big enough to be useful for a full BFR passenger flight, assembling) a space station which is roomy enough and well equipped enough for every passenger to stay a while.

Maybe there are some illnesses where you can benefit from being in zero g

This is a mainstay of classic Sci-Fi, but so far IIRC every effect we've found has been in the opposite direction. Zero G is like anti-exercise for the human body, like being bedridden.

and you can have your hospital outside of any jurisdiction.

This makes more sense, and (unlike the rest of my huge digression) gets straight at the heart of the post here, but I don't think orbit adds any extra sense to the idea. There are a hundred little jurisdictions around the world who would happily issue the medical equivalent of a "Flag of Convenience" and waive whatever regulations you think are stupid in return for getting heavy investment and medical tourism. Even if there weren't, and you had to go outside every jurisdiction, international waters a hundred miles offshore would have the (questionable!) legal status you're looking for and would still be cheaper and more convenient than international space a hundred miles up.

I have a possibly stupid question. I know health care in the United States is totally screwed up; how does it compare to India, China, and Russia?

At least the first two chapter links are broken (currently). It says

Sorry, we couldn't find what you were looking for.

I'm with Simplicio, since my definition of "evil" is "following one's individual incentives in situations where you know you are perpetuating a drastically suboptimal equilibrium." If you're a scientist, it's not moral to pursue a maximally impressive career at the cost of ignoring a bunch of work that you think really needs doing. (If you don't know it needs doing, then "stupid" applies.)

I think highlighting this as a moral imperative and appealing to people's conscience is a reasonable approach to attacking these problems, since it seems like more or less how we made it up to the equilibria we're at now.

If I'm understanding you correctly, you don't think people who can't individually affect the equilibrium are evil? Scientists who would be outcompeted, and therefore unable to do science, if they failed to pursue a maximally impressive career seem an example of this. If they're good (in terms of both ability and alignment), there's some wiggle room in there for altruism when it's cheap, but if err too far from having an impressive career, someone else, probably someone not making small career sacrifices for social benefit, gets the attention and funds instead, thereby reducing the total amount of good being done.

I'd be interested in some concrete examples of appealing to people's conscience getting us less-bad equilibria so I can better understand what you're getting at. Is the scientists who all resigned from the board of an Elsevier-owned journal and started their own an example?

Also interested in your thoughts where we're in a suboptimal equilibrium that we're trying to get out of and there's 2+ optimal ones (in the sense that there's no other equilibrium which is a Pareto improvement), but each competes with the other. For instance, suppose it's several years ago and we agree that it's better that gay couples have the same right to legal marriage as straight couples, but the two better equilibria are (1) elimination of marriage as a legal institution and (2) extension of marriage to gay couples, which is good for gay couples who want the legal benefits of marriage and less good for those who don't want marriage but suffer from e.g. less favorable tax treatment. (I'd really like a better example for this, especially after the examples EY gave, but lack a large, responsive group of fb followers I can get to brainstorm examples for me.)

There are definitely a lot of people who are competing on the margin, for whom deviating from their individual incentives would cause them a lot of suffering and eliminate other ways they can do good in the world; I really can't fault them for not making those sacrifices.

However, I think there are many people who only have to make relatively small personal sacrifices to this end. For example, one link in the footnotes is about the failure of companies to reward interviewers for doing a good job. Many -- most -- interviewers are people with established careers and non-zero political capital. If so, you may very well have the slack to just spend extra time doing a good job anyway, and suck up the fact that your other work will slow down a little. If you think that doing a better job hiring people will have an important impact in the world, and you have the ability to sacrifice a relatively small slice of your career to do it, you should do it. It's not OK to do a bad job just because it's maximizing your self-interest.

Note that many of the most powerful decision-makers in our society are also the most privileged; they are wealthy, have high status, and have personal attributes that make their lives easier. (AFAIK this is least evident in the academic community -- but consider, for example, celebrities, rich capitalists, and many upper-middle class professionals.) My experience is that by and large, those people have the latitude to largely blow off bad incentives while incurring mild personal suffering at best -- e.g. they might risk a single job in a world where many similarly good jobs await them, or risk some money from a pile of money that is already not getting them much more marginal utility per dollar. When those people still choose to maximize their own self-interest? That's as close to evil as you can get.

(I get the sense that Eliezer thinks that this "free energy" gets eaten up very quickly by other actors who will outcompete the person who is making sacrifices. I just don't think that's the case, because the world has large variations in ability and situation. If you are an impressive manager with an impressive resume, and you sacrifice 50% of your potential impressiveness for the rest of your life to do idiosyncratic altruistic things, and nobody else does, that still leaves you more impressive than a lot of your competition. You can still go get powerful jobs where you can continue to do altruistic things all day.)

I'd be interested in some concrete examples of appealing to people's conscience getting us less-bad equilibria so I can better understand what you're getting at. Is the scientists who all resigned from the board of an Elsevier-owned journal and started their own an example?

In that space, people uploading things to Sci-Hub from their academic accounts is another example that is more along the lines of what I'm thinking of -- something you can do personally when you see an obviously better equilibrium that you can make a sacrifice to move towards. The kinds of stuff I mean:

  • Standing up as a representative of an oppressed group, instead of keeping your head down (e.g. think civil rights activists.)
  • As above, spending time in your professional work to maximize quality metrics that you know are important but under-rewarded. When you see a professional code of conduct or standards of quality that actually make
  • "Consumer activism", i.e. supporting companies and products that are taking into account important but otherwise under-rewarded externalities that you care about.

Also interested in your thoughts where we're in a suboptimal equilibrium that we're trying to get out of and there's 2+ optimal ones (in the sense that there's no other equilibrium which is a Pareto improvement), but each competes with the other.

That's really hard. I don't have any very good plan for those.

I agree with most of this. The fight has to start somewhere. It is almost certainly not a sufficient condition for escape, but definitely a necessary one, to have people be aware that other ways of doing things might exist (at the risk of sounding a little Simplico here, I think that a large part of the problem is that many people for whatever reason don't imagine or contemplate that things could potentially be different than they are (in either worse or better directions).

I also agree that the free energy issue is heavily muddled by wildly varying individual abilities and competencies. So I'm a scientist that, to be blunt, is significantly more talented and competent at my field than most of my peers. Unlike them, I haven't sold my soul to the system, but I'm still matching them, or in some cases exceeding them for publication output. Could I 'sell out' and gain a lot of fame and prestige and stuff? Maybe, but why should I care about that? My preferences don't consider that valuable. I still very much have to choose my battles, but it will be more of a loss for me if I end up getting a presitigous position while having what I'm doing become decoupled from all its meaning than being pushed out and not allowed to continue. In the second case, I'll be able to walk down a different road and do something else with my life. Maybe that day will come eventually, maybe it won't. Either way I'm going to continue as I am, trying to do my research for its actual own intrinsic sake, because I could never be personally satisified otherwise. I care a little more about the possibility of actually getting out of the rut one day than about hollow personal advancement. But as a warning, in order to take this path, you do have to be significantly better than average to survive

(as an aside, I think it's considered rude to talk about how smart you are or whatever because that's kind of an advanced stage lemon problem. People would say that regardless. So it's up to you whether to believe my assertions of competence. Only I can (maybe) know their veracity. But do keep in mind that I would sensibly keep that all to myself for the sake of not sounding like an arrogant ass (which I know I do right now) if it weren't drectly and specially relevant to the discussion at hand).

short version: you can defy the equilibrium in certain ways if you don't care about the 'rewards' and 'punishments' of that particular aspect.

What is the balance, though?

If I don't have a job, my children will starve. If I want to keep a job, I need to make some concessions that in part perpetuate one or more suboptimal equilibria. This is true no matter what my job is.

Yes, everything is terrible. But it seems like, if you're writing a book and discover something like the Omegaven story, it might be worth writing a blog post just about that and seeing if it can get some publicity via social media? (I settled for resharing the 2013 NBC article.)

So there’s no way for your planet to try different ways of doing things, anywhere. You literally cannot run experiments about things like this.

Well, there's 200 countries, so there are 199 experiments going on. But apart from that. no.

Maybe “Nobody ever got fired because of the later performance of someone they turned down,” but they might wind up in one of those lists that circulate all over the Internet (and by photocopied and faxed documents in pre-Internet days) of famous people who got humorously rejected, like the Beatles being turned down by a record executive because guitar music was going out of fashion or something. Of course, many of the things on those lists are probably exaggerated or even made up out of whole cloth.