We need a new paradigm for doing medicine. I make the case by first speaking about the problems of our current paradigm of evidence-based medicine.
The status quo of evidence-based medicine
While biology moves forward and the cost of genetic-sequencing dropped a lot faster than Moore's law the opposite is true for the development of new drugs. In the current status quo the development of new drugs rises exponentially with Eroom's law. While average lifespan increased greatly about the last century in Canada the average life span at age 90 increased only 1.9 years over the last century. In 2008 the Centers for Disease Control and Prevention reported that life expectancy in the US declined from 77.9 to 77.8 years. After Worldbank data Germany increased average lifespan by two years over the last decade which is not enough for the dream of radical lifespan increases in our lifetime.
When it costs 80 million to test whether an intervention works and most attempts show that the intervention doesn't work we have a problem. We end up paying billions for every new intervention.
Eric Ries wrote "The Lean Startup". In it he argues that it's the job of a startup to produce validated learning. He proposes that companies that work with small batch sizes can produce more innovation because they can learn faster how to build good products. The existing process in medicine doesn't allow for small batch innovation because the measuring stick for whether an intervention works is too expensive.
In addition the evidence-based approach rests on the assumption that we don't build bespoke interventions for every client. As long as a treatment doesn’t generalize about multiple different patients, it’s not possible to test it with a trial. In principle a double-blind trial can't give you evidence that a bespoke intervention that targets the specific DNA profile of a patient and his co-morbidity works.
The ideal of prediction-based medicine
The evidence-based approach also assumes that practitioners are exchangeable. It doesn't model the fact that different physical therapist or psychologists have different skill levels. It doesn't provide a mechanism to reward highly skilled practitioners but it treats every practitioner that uses the same treatment intervention the same way.
Its strong focus on asking whether a treatment beats a placebo in double-blind studies makes it hard to compare different treatments against each other. In the absence of an ability to predict the effect sizes of different drugs with the literature the treatment that wins on the market is often the treatment that's best promoted by a pharmaceutical company.
How could a different system work? What's the alternative to making treatment decisions based on big and expensive studies that provide evidence?
I propose that a treatment provider should provide a patient with the credence that the treatment provider estimates for treatment outcomes that are of interest to the client.
If Bob wants to stop smoking and asks doctor Alice whether the treatment Alice provides will result in Bob not smoking in a year, Alice should provide him with her credence estimation. In addition Alice’s credence estimations can be entered in a central database. This allows Bob to see Alice’s Brier score that reflects the ability of Alice to predict the effects of her treatment recommendations.
In this framework Alice’s expertise isn't backed up by having gotten an academic degree and recommending interventions that are studied with expensive gold-standard studies. Her expertise is backed by her track record.
This means that Alice can charge money based on the quality of her skills. If Alice is extremely good she can make a lot of money with her intervention without having to pay billions for running trials.
Why don't we pay doctors in the present system based on their skills? We can't measure their skills in the present paradigm, because we can't easily compare the outcomes of different doctors. Hard patients get send to doctors with good reputations and as a result every doctor has an excuse for getting bad outcomes. In the status quo he can just assert that his patients were hard.
In prediction-based medicine a doctor can write down a higher credence for a positive treatment outcome for an easy patient than a hard patient. Patients can ask multiple doctors and are given good data to choose the treatment that provides the best outcome for which they are willing to pay.
In addition to giving the patient a more informed choice about the advantages of different treatment options this process helps the treatment provider to increase his skills. They learn about where they make errors in the estimation of treatment outcomes.
The provider can also innovate new treatments in small batches. Whenever he understands a treatment well enough to make predictions about its outcomes he's in business. He can easily iterate on his treatment and improve it.
The way to bring prediction-based medicine into reality
I don't propose to get rid of evidence-based medicine. It has its place and I don't have any problem with it for the cases where it works well.
It works quite poorly for body work interventions and psychological interventions that are highly skill based. I have seen hypnosis achieve great effects but at the same time there are also many hypnotists who don't achieve great effects. In the status quo a patient who seeks hypnosis treatment has no effective way to judge the quality of the treatment before he's buying.
A minimal viable product might be a website that's Uber for body workers and hypnotists. The website lists the treatment providers. The patient can enter his issue and every treatment provider can offer his credence of solving the issue of the patient and the price of his treatment.
Before getting shown the treatment providers, a prospective patient would take a standardized test to diagnose the illness. The information from the standardized test will allow the treatment providers make better predictions about the likelihood that they can cure the patient. Other standardized tests that aren’t disease specific like the OCEAN personality index can also be provided to the patient.
Following the ideas of David Burn’s TEAM framework, the treatment provider can also tell the patient to take tests between treatments sessions to keep better track of the progression of the patient.
When making the purchasing decision the patient agrees to a contract that includes him paying a fine, if he doesn’t report the treatment outcome after 3 months, 6 months and 1 year. This produces a comprehensive database of claims that allows us to measure how well the treatment providers are calibrated.
Various Quantified Self gadgets can be used to gather data. Many countries have centralized electronic health records that could be linked to a user account.
The startup has a clear business model. It can take a cut of every transaction. It has strong network effects and it's harder for a treatment provider to switch because all his prediction track record is hosted on the website.
Thanks to various people from the Berlin Lesswrong crowd who gave valuable feedback for the draft of this article.
Let's look at schools. Having a masters degree in teaching doesn't result in the teachers students getting higher grades on standardized tests. Unions still press schools to pay people who have a masters degree in education more money and mostly succeed at opposing pay-for-performance.
Education is a good example of a field where teachers are payed for useless training instead of being payed for producing outcomes for students.
Bill Gates suggest in his TED talk:
Given that citing a TED talk as a response to a scientific paper is a bit bad form, I added a question on skeptics to verify the claim.
I think part of the problem of the cited study is likely that it's done on data from before no-child-left-behind and the efforts of the Gates Foundation. It's also simply possible to find groups of teachers where there's little variance in teaching skill, that doesn't mean that differences don't exist on a larger scale.
There are treatment that require high skill to administer like Brain surgery and treatments that require less skill like handing over a pill. For the high skill tasks I do think that variance in patient outcomes would decrease.
But even for simply taking a pill a doctor can spend five minutes to hand over the pill or he can spend an hour to talk through the issue with the patient and make sure that the patient has TAP's to actually take the pills according to the schedule.
Currently there's no economic reason to spend that hour. It can't be billed to the insurance company. Even if currently everybody spends five minutes for such a patient you would suddenly get variance if you would start to pay some doctors by the outcome instead of simply paying them per visit.
I don't think that large sample sizes are everything. I think you can learn a lot by looking very carefully at the details of single cases.
Additionally the person who's the best in city X at treating disease Y for class of patients Z might get more than 40 patients of class Z with Y because everybody wants to be treated by the best.
If all those expensively trained doctors achieve the same outcomes there's also a question of why we limit their supply as strongly as we are doing it at the present by forcing to them to have the expensive training. In that case the way to go would be to get people with cheaper training to compete in the market of the highly trained doctors. In the absence of performance tracking this won't be possible because the expensively trained doctors have more prestige.
That doesn't surprise me. Training and degrees are easily observable; outcomes (or, rather, how much of an outcome is attributable to each teacher) are not. It's harder to pay people based on something less observable.
Thanks!
But I'm not sure it matters. What Bill Gates says in that quotation might well be consistent with what I wrote — it's hard to be sure because he's quite vague.
For instance, he says that a teacher in the top quartile increases their class' performance "by over 10 percent in a single year". I can believe that, but maybe all it means is that a top-quartile teacher increases their class' scores by 11% a year, while a bottom-quartile teacher increases their class' scores by 9% a year. That would hardly refute the idea that teacher effects are small on average!
Maybe. I did a bit more searching on Google Scholar but didn't uncover a more recent review article with relevant statistics. (I did find one study of a thousand students in HSGI schools in "school year 2013-2014", which found that 12% of the variance in GPA was at the teacher level.)
However, while the study I cited uses only pre-2004 data, I don't see much reason to think a newer study would reveal a big increase in teacher-level variance in outcomes. The low proportion of variance attributable to teachers has been true, as far as I know, for as long as people have investigated it (at least 46 years). I'm doubtful that an act which demands high-stakes testing, improvements in average state-wide test scores, and state-wide standards for teachers has changed that, or that a charity has changed that.
It's possible, but I don't see evidence that the paper I cited has this flaw. Its new analysis was based on about 100 Tennessee schools included in Project STAR, and the earlier analyses it summarized (table 1) used "samples of poor or minority students", "nationally representative samples of students", and "a large sample of public school students in Texas".
Fair point — my thought experiment only addresses diagnosis and treatment recommendation, not treatment administration. I think there would be more doctor-level variation among the latter...although not much more in absolute terms.
Probably true in most places.
Agreed that you'd get more variance, but I suspect it wouldn't be much more (subject to this hypothetical's exact details).
This is certainly true. I've seen too many stories of people with rare genetic conditions and other diseases successfully working out aetiologies to think otherwise. But those are unusual cases where people tended to invest lots of effort into figuring their cases out. I don't see them as signs that we can improve the quality-to-cost ratio of healthcare in general by looking very carefully at the details of each case.
Sure. But the ratios in my example are more important than the exact numbers.
A good question, and a good answer to the question!
It'd be immediately possible in the US: eliminate de facto immigration barriers for foreign doctors. Those barriers are, I'd guess, lower in other developed countries (hence why doctors in the UK, Germany, Canada, etc. earn less than US doctors) but I expect doctors' salaries there could also be reduced a bit by further relaxing immigration restrictions for foreign doctors.
Another option is for patients to go to the cheaper doctors: medical tourism.